fix(api): suppress stuck-instance alerts for hosts in maintenance mode

[kdhulipala-wq]

Description

Stuck Assigned-substate machines sometimes take days to resolve, and operators put them into maintenance mode to silence the alert. The existing setup does not suppress alerts for these machines when they are put into maintenance mode, and we are still receiving the PD alerts.

This PR makes SetMaintenance::Enable also write ExcludeFromStateMachineSla — matching what the admin-cli InternalMaintenance template has been doing, so state_sla() short-circuits to no_sla() and a host in maintenance stops contributing to stuck-instance alerts regardless of which state or substate it's in.

Scope

The motivating case is Assigned-substate machines paging via stuckInstanceCritical, but the fix is applied at the state_sla() layer, which means it applies to all machine state-SLA trackingm, not only the Assigned family.

• Alerts : stuckInstanceCritical and manyStuckInstancesCritical already filter on {state="assigned"}, so the on-call behavior is exactly what the bug report asked for.
• Grafana / observability : panels that query forge_machines_per_state_above_sla with broader filters (e.g. state!="assigned") will also stop counting a machine while it's in maintenance.

Type of Change

• Add - New feature or capability
• Change - Changes in existing functionality
• Fix - Bug fixes
• Remove - Removed features or deprecated functionality
• Internal - Internal changes (refactoring, tests, docs, etc.)

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed
• No testing required (docs, internal refactor, etc.)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[krish-nvidia]

Just semantics, but this will also make time_in_state_above_sla false for non-assigned machines. There’s a Grafana dashboard panel that uses forge_machines_per_state_above_sla{fresh="true", state!="assigned"}, so putting one of those machines into maintenance mode will now remove it from that view too.

I think that’s expected behavior, but maybe update the PR description/comments to call out that this applies to all machine state SLA tracking, not just assigned machines. The unit test also doesn’t create an instance, so it’s already validating this broader behavior 😄

[kdhulipala-wq]

Updated!

[ajf]

/ok to test fe74b65