Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added images/0877/impact-effort.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/0877/impact.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/0877/system.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/0877/triage.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
125 changes: 125 additions & 0 deletions text/0877-automated-issue-triage.md
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that community reviews would essentially be phased out / no longer serve their role?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point. i did not consider community reviews initially but you're right that this proposal disregards them.

in truth, i don't think community reviews has been a popular initiative besides a select few community members who are willing to spend some of their time reviewing. i want to still encourage and reward this behavior, but i'm not sure if the current community review setup is the best way to do that. @badmintoncryer if you have any ideas, i'm happy to hear them! otherwise i'll keep thinking on this.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your response.

As you pointed out, there are not many reviewers who actively engage in community reviews, and I feel that we are not achieving much throughput at the moment. I believe one of the reasons for this is that the benefits and motivation for conducting reviews are difficult to feel.

If this RFC is formalized, I expect the bar for reviews to rise and the number of community-sourced PRs that get merged to decrease. In such a situation, I felt that if PRs approved by community reviewers were made eligible for review without requiring five reactions, it would make it easier for reviewers to get their desired features merged, which in turn could help boost motivation to participate in reviews.

This is just one idea, but I will also talk with community reviewers around me to see if there are any other good suggestions.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I felt that if PRs approved by community reviewers were made eligible for review without requiring five reactions

+1
PRs approved by a community reviewer should be eligible for maintainer merge regardless of the issue's interaction count. This would serve as a practical mitigation for the strict 5-reaction threshold, reduce the maintainer review burden, and give community reviewers a stronger incentive to stay engaged.

I believe one of the reasons for this is that the benefits and motivation for conducting reviews are difficult to feel.

On the motivation side, moving the community reviewer list from the wiki into a dedicated file in the repository (like CONTRIBUTORS.md) could help. Not many people check the wiki, so most contributors probably don't even know community reviewers exist. More visibility would naturally lead to more recognition and motivation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the point in reviewing code if its never going to end up in the library anyway. The fundemental issue is not how its labelled.

Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# Automated Issue Triage

* **Original Author(s):**: @conroyka
* **Tracking Issue**: #877
* **API Bar Raiser**: @mrgrain

> This RFC does not follow the typical RFC framework because it is not a technical design.
> I still want to work out in the open though!

## Why

Contributing to the CDK has long been messy, tedious, convoluted, etc. Meanwhile agentic AI is
reshaping the landscape of open-source contributions as it lowers the barrier to contribute
PRs to aws-cdk. We are seeing (and encouraging) more AI-generated contributions but this comes with
significant challenges -- increasing throughput of PRs is not always a net-positive, and ignores the
bigger picture of the code lifecycle (reviewing and maintaining in perpetuity).

We also have 2300+ open issues. We don't actually _want_ contributions to solve all these issues;
many open issues are _unwanted_: some combination of low impact and high effort (here "effort" means
both cost to create the PR and cost to maintain the changes). The diagram below conveys what we want
to support in the pre-AI era:

![Graph with effort on the x-axis and imapct on the y-axis](../images/0877/impact-effort.png)

However, some of our assumptions don't hold true anymore.
First of all, effort level triage encapsulates developer effort only (which is rapidly minimized
by agentic AI) and does not consider review and maintenance effort (which is what we
care about). Second of all, initial triaging of priority is just a best guess, so we hedge:
if you author a PR for a `p2` contribution, we wont promise a review but maybe you will get one
eventually. This is inefficient because the decision on whether an issue should be fixed at all
happens only after a PR is created. And this consideration is left subjective to the reviewer,
which causes friction when we decided an approach wasn't worth the effort in the review phase.

Ultimately, this proposal aims to increase efficiency: an automated triage process with a clear
set of rules provides clarity over what issues we want solved, and limits churn on community
contributed PRs -- we will have high confidence that all PRs that come in are solving an issue
that has [Demonstrated Customer Impact](#demonstrated-customer-impact).

## Proposal

We will:

1. Fully automate triaging with new labels, and an appeals process via community input.
2. Be extremely clear on what subset of open issues welcome community contributions.
3. Meaningfully lower the open issue count in the aws-cdk repo by closing unwanted issues.

### Priority Labels

- `regression`: This is _actively_ worked on by a CDK maintainer at a fast pace. `regression` replaces the `p0` label exactly.
- `maintainer-response`: An issue from a lower priority tier that has been bubbled up to review.
- `maintainer-claimed`: Important issues (i.e. bugs without a workaround) that CDK Maintainers are
actively working on. After 14 days of inaction these issues will be upgraded to `maintainer-response`.
- `accepting-prs`: Bugs/features that have [Demonstrated Customer Impact](#demonstrated-customer-impact). Indicates that the CDK team
is happy to review and maintain.
- `incubating`: Bugs/features that we do not want to support without better signals from our customers.
All incoming issues with workarounds are triaged here to start. Issues that subsequently show
[Demonstrated Customer Impact](#demonstrated-customer-impact) are upgraded to `accepting-prs`.
- `unwanted`: `incubating` issues that have been human-reviewed and closed.

### Effort Labels

`design-required`: Feature requests that were previously labelled `effort-large`. No one, not the
community, maintainers, or AI should action on these issues without an approved RFC or design artifact.
`automatable`: Issues that are straightforward and can be automated. This grouping will expand
over time. `automatable` issues will kick off a human-in-the-loop AI cycle to create a high-quality
PR with minimal lead time. At the start, `automatable` issues are _issues that do not require
testing_ -- think updates to [enums](https://github.com/aws/aws-cdk/pull/36798), [region info](https://github.com/aws/aws-cdk/issues/36880), docs, etc.
`good-first-issue`: This is an open-source standard that I would like to keep as an ad-hoc
curated list of what we think is the easiest selection of available and wanted issues.

### Triage

Triaging is simple and well-defined, allowing for a fully automated triage process for CDK and
other AWS open-source projects that adopt this proposal. Issues can be triaged immediately and
maintainers can spend their time elsewhere.

Notably, our initial triage process results in only a subset of labels being used. Other labels
require additional time windows. See [Upgrades and Downgrades](#upgrades-and-downgrades) for details on
how issues get upgraded.

![Triage Workflow](../images/0877/triage.png)

### Upgrades and Downgrades

1. All incoming issues start as `incubating`. Issues that [demonstrate customer impact](#demonstrated-customer-impact)
are upgraded to `accepting-prs`.

2. A `maintainer-claimed` issue is awaiting maintainer action, a phrase which here means providing
a workaround or creating a PR. After 14 days of inaction, `maintainer-claimed` issues upgrade to
`maintainer-response`.
Comment on lines +88 to +90
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean community members should not submit PRs for maintainer-claimed issues? It would be helpful to state this explicitly to avoid wasted effort on either side, e.g., a community contributor spending time on a fix that conflicts with work a maintainer has already started.


3. `incubating` PRs are eligible to be closed as `unwanted` with a reason provided as a comment.
Closure comes with a human review but AI will assist the decision making. AI helps us be more efficient
when it comes to surfacing issues we may not want, based on subjective criteria (duplicate of previously
closed issue, feature request misidentified as a bug, not the path we want to go down).

### Pull Requests

Pull requests will inherit the labels of the issue they address. Pull requests without a linked issue
are by default labeled `incubating`. We will have heavy-handed communication in the Contributing Guide
Comment on lines +99 to +100
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For PRs that are not linked to an issue, would they be promoted to accepting-prs if the PR itself receives more than 5 reactions?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this should be the case. and i hear you that for this proposal to work, we need to make it very clear that reactions on issues are being used to determine impact in this manner. it will take some time for the community to adjust to this new weight that interacting with issues holds

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will take some time for the community to adjust to this new weight that interacting with issues holds

@kaizencc I reckon the proposal should have a plan for that. We know the RFC process doesn't fit here, but for technical designs we always have a rollout plan. this seems similar.

and in the PR description template that `incubating` PRs will not be reviewed. After 28 days,
`incubating` PRs are auto-closed. Critically, `incubating` PRs are `incubating` because they have been
identified as not worth review effort and maintenance effort, so we will ignore them.

### Demonstrated Customer Impact

An issue must demonstrate customer impact during an incubation period of 28 days to upgrade to `accepting-prs`.
To demonstrate customer impact, 5 unique individuals must react to, comment, or link the issue. This number
is a proposal but derived from data from 1000 recent issues:

![Impact](../images/0877/impact.png)

## Epilogue

There are clear expectations in this model. `maintainer-claimed` issues are claimed and actively worked on by
CDK maintainers. `accepting-prs` issues are eligible for community contributions. No one should contribute a
solution to `incubating` issues.

![Overall System](../images/0877/system.png)

I want to be clear — these are intentionally rigid rules for triage automation to follow. Triage is inherently
a best-effort activity, and with the right guardrails is ripe for automation. Separate to the triage process
humans and additional tools should intervene where appropriate to move issues from one bucket to another.

The goal is to save triage time and be transparent with the community on what is a good idea to work on.