Retriggering without in-tree-actions

[ahal]

There's a draft RFC up for this, but I don't think we're quite at the point where we're ready for an RFC just yet as there are some issues to work out. Let's get consensus on the high level approach here and then write out the RFC to tackle the implementation details.

How Retriggers Currently Work

Currently when a developer re-triggers, they click a button in Treeherder, which fires an in-tree-action hook task. The template for this in-tree-action hook is the contents of the repo's .taskcluster.yml file, which was downloaded by fxci-config at deploy time. The in-tree-action hook template is rendered from context that conforms to a trigger_schema and that (presumably?) gets provided by Treeherder. It appears that Treeherder consumes the actions.json artifact from the Decision task, and includes this context as part of the trigger_schema.

This in-tree-action hook task then:

1. Clones the repo
2. Executes the relevant action using taskgraph action-callback

For retriggers, this action task typically takes ~1-2 minutes if there is a hot checkout cache. But it can take ~4-5 minutes if it needs to clone.

Problem

There are two problems we'd like to solve here:

1. Improve performance of re-triggered tasks
2. Improve perception of sluggishness from developers

Performance wise, saving ~1-2 minutes off a test task that takes ~20-40 minutes is nice, but not a huge win imo. But the felt impact of needing to wait for an "rt" task to complete makes it feel much more impactful than it likely is. This felt impact is magnified by Treeherder's ingestion delay. First, developers need to wait for rt to show up. Then they need to wait 1-5 minutes for it to finish. Then they need to wait for their actual task to show up.

Whether it's worth the effort to reduce the felt performance or focus solely on actual performance is not an answered question imo. However, I do wholeheartedly agree that it would be much nicer overall if retriggers were snappier. Gecko test task retriggers in particular are like 95% of all action tasks in fxci (I made that up, but it's probably close), so focusing in on these would have outsized benefits on Taskcluster's perception.

Possible Solutions

The root of the issue is that the in-tree-action task clones the repo regardless of what action is being re-triggered. What we want is some kind of logic that goes like this:

if action_type == "retrigger" and taskdef["extra"]["retrigger"] is True:
    call the TC create task API
elif action_type == "retrigger" and taskdef["extra"]["retrigger"] is False:
    call the TC rerun API
else:
    normal action task flow of cloning repo + running taskgraph action-callback

Here are three possible places we could put this logic:

1. In .taskcluster.yml - We could likely solve this without any fundamental changes by modifying the .taskcluster.yml file to have completely different behaviour if the action name is "retrigger". The downside is that we'd need to implement this change in every project we want this behaviour, and it would make the .taskcluster.yml files even more complicated.

2. In the in-tree-action hook generation code - Another approach might be to have the code that makes these hooks "wrap" whatever is in the repo's .taskcluster.yml files in some outer JSON-e logic. This would solve the downsides of 1, but we'd be "concatenating" JSON-e, which theoretically should work fine I think, but might be messy and will definitely be confusing to future maintainers of this.

3. In separate global hooks - Finally, we could create brand new global hooks in the fxci-config repo specifically for retriggering (i.e, move them out of the "generic" actions). It's unclear to me if there are still scenarios where we'd need to clone the repo, or if every task can boil down to a binary "should rerun or not". If it's the former, this becomes a bit more complex, but probably still do-able. Another downside here is scopes, but we can likely mitigate this by generating one hook per trust domain / level, it would just require some new code logic in ciadmin.

Overall, I like option 3 the best if it's feasible. If it's not feasible, I like option 2 the second best. If we want a quick win and are happy for this feature to be Gecko-only, option 1 might even be fine.

[ahal]

Currently this approach assumes we'll just stuff whether a task should be retriggered or not into the "extra" field, which is probably fine.. But maybe it's also worth talking about solving this as a feature of the Taskcluster platform. E.g, make it an actual property of the task definition, then have the queue service take appropriate action in the actual API call.

[bhearsum]

In general at a high level, I fully support special casing retriggers (and perhaps other things, like cancel/cancel all?) to go directly through taskcluster APIs instead of require a checkout to execute. Other actions (notably rerun, release-promotion, and other CoT-impacting things) are a different story of course, and I think you've pretty clearly put them out of scope here already.

If I understand the overall idea correctly, none of the proposed solutions would require granting extra scopes to developers, which I think is good. (The taskcluster APIs would still be called in the decision/action tasks, just more directly than they are now.)

I'm generally not super high on options 2 or 3 due to the added complexity. Any solution that requires wrapping .taskcluster.yml contents in an outer layer is going to hugely hurt our ability to debug things.

My inclination is to move forward with Option 1 just in Gecko for now, and see how that goes. If it works well, we can choose to carry that option over to other repositories, or consider other ways to share the solution. If we discover issues with it, we'll have to re-evaluate.

I think I'm less concerned than you/others about adding more to .taskcluster.yml that needs to be duplicated around. It's not ideal, of course, but I'd much rather we more generally solve the problem of "sharing complex json-e" than try to special case a solution for this.

Perhaps also of note is that most repositories don't take very long to clone. Besides Gecko and Comm, I'm not sure the cloning is a big deal elsewhere. I'm also not sure if retriggers are used often elsewhere, so perhaps we wouldn't even bother applying this fix to most other repositories?

[JohanLorenzo]

Thanks for opening this issue, @ahal ! I agree we don't have a clear direction about how we want to tackle that problem. I didn't think of optimizing the action tasks themselves so thank you for bringing them up!

I want to propose a couple more options that complement what you've outlined. For most retriggers, we don't need action tasks at all.

Option 4: Create a new /retrigger endpoint to the queue and let clients decide what to do

Add a POST /task/:taskId/retrigger endpoint to the Queue service. taskgraph can add retrigger-method tag for each task based on whether it's safe to retrigger directly (leaf task, no CoT requirements, not on scriptworker pools). The endpoint would:

• Fetch the original task
• Create a new task with a new taskId
• Adjust timestamps relative to current time
• Preserve all other properties (routes, dependencies, scopes, payload)
• Require a new scope: queue:retrigger-task:<schedulerId>/<taskQueueId>

All clients, like the Taskcluster UI and Treeherder would check this tag and call either the direct endpoint or trigger the action hook. This keeps the backend simple but adds some routing logic to consumers. There's already some of this routing logic in the Taskcluster UI, though.

Option 5: Centralize the decision of spawning an action task in the Queue.

Similar to option 4, but when doing aPOST /task/:taskId/retrigger, the queue checks the retrigger-method tag does what clients do in option 4. It's cleaner for consumers (one call), but the Queue service gets more complex and gets more tighter dependencies to:

• taskgraph (because of the new tag and the dependency on `actions.json)
• the hook service (in order to spawn the action task)
• the content of .taskcluster.yml as you highlighted above.

I put option 5 for the sake of the argument but I believe it's too big of a change. Option is heavier than options 1 to 3 but it gets rids of retrigger action tasks whenever possible.

[ahal]

My inclination is to move forward with Option 1 just in Gecko for now, and see how that goes. If it works well, we can choose to carry that option over to other repositories, or consider other ways to share the solution. If we discover issues with it, we'll have to re-evaluate.

This makes sense to me. This probably isn't even RFC worthy, just needs someone to try it out.

I want to propose a couple more options that complement what you've outlined. For most retriggers, we don't need action tasks at all.

Right, the pseudo-code I wrote could go in Treeherder as well. But then you need to give it blanket create-task scopes, which scares me a bit.

Question: Is it actually important to get rid of action tasks? Or is overall e2e time the only thing that matters? Because if the issue is "Developers don't want to see 'rt' tasks showing up", we can just not show them on Treeherder :p

[bhearsum]

I want to propose a couple more options that complement what you've outlined. For most retriggers, we don't need action tasks at all.

Right, the pseudo-code I wrote could go in Treeherder as well. But then you need to give it blanket create-task scopes, which scares me a bit.

Question: Is it actually important to get rid of action tasks? Or is overall e2e time the only thing that matters? Because if the issue is "Developers don't want to see 'rt' tasks showing up", we can just not show them on Treeherder :p

And to add on to this: we can do things to reduce the time (both pending and running) that action tasks take, and the perceptions issues of things not showing up in treeherder. We've already made dramatic improvements to decision task times, some of which has likely improved action tasks already. I've been looking at the latency and perceived latency of task status changes between taskcluster and treeherder, and I suspect there's a lot of improvement we can make on that side as well.

IMO it would worthwhile to wait to see if such improvements get us far enough (which will benefit more than just action tasks, so we should do them anyways...) before we make such a sweeping change.

[ahal]

If we're planning on solving the minCapacity issue at the work week, that's another reason not to change things up too much right now. We could have a dedicated pool for them with minCapacity set such that tasks almost always start within seconds

[JohanLorenzo]

Discussed with @ahal over a Zoom call. I agree, let's see the impact of the minCapacity fix to gauge whether we need to revamp the retrigger actions.

[fqueze]

Performance wise, saving ~1-2 minutes off a test task that takes ~20-40 minutes is nice

I think there's a specific case where the performance win would be significant: when working on debugging an intermittent test failure, the workflow is to push to try for that specific test path, and then the actual test task is very short, typically less than a minute. While most retrigger tasks take less than a minute, some clone the repository and take ~4 minutes.