Add flag live migration by jose-caballero · Pull Request #459 · stfc/st2-cloud-pack

jose-caballero · 2026-06-29T09:01:36Z

Description:

Special Notes:

Submitter:

Have you (where applicable):

Added unit tests?
Checked the latest commit runs on Dev?
Updated the example config file(s) and README?

Reviewer

Does this PR:

Place non-StackStorm code into the lib directory?
Have unit tests for the action/sensor and lib layers?
Have clear and obvious action parameter names and descriptions?

This flag is needed for both cases: - when the entire hypervisor.drain workflow is executed - when only the server.migrate action is executed Because of that, we need to add the flag in the YAML configuration file for both Actions (hypervisor.drain and server.migrate), and also in the Orchesta YAML configuration that connects those two Actions. The behavior of this flag is as follows: * only affects the migration of ACTIVE Servers. If the Server is already SHUTOFF, an unconditional Cold Migration will be performed * for ACTIVE Servers * if the flag is set to True (default value), then a Live Migration will be attempted * if the flag is set to False, a Cold Migration will be performed

codecov · 2026-06-29T15:09:46Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.19%. Comparing base (509c54b) to head (bdf2fa7).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #459   +/-   ##
=======================================
  Coverage   99.19%   99.19%           
=======================================
  Files         112      112           
  Lines        2726     2737   +11     
  Branches      338      339    +1     
=======================================
+ Hits         2704     2715   +11     
  Misses         19       19           
  Partials        3        3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

DavidFair · 2026-06-30T12:14:49Z

Notes from testing:

To fix:

Attempting to live_migrate a GPU will snapshot first (could take hours) then will fail as it's blocked, need to check if the migration is permitted before doing any work
This does a live snapshot by default which requires qemu-ga. I think we should force an offline snapshot for safety to handle cases where qemu-ga isn't working and blocks a live migration (see below)
If the source is disabled and the destination has no slots the ST2 action will sit pending despite it going into error on server migration list. I think there's a bug on detecting a migration (not server) going into error state and this caught me out several times and needs fixing before we do this for real

To note (gotchas):

When no space is available VM migrates back to same HV (no shutdown triggered) if it's not marked disable
VM with qemu-guest-agent uninstalled / offline gives quiescing instance: Error from libvirt while quiescing [Error Code 86] Guest agent is not responding: QEMU guest agent is not connected. but will snapshot still giving potentially partially written images. If we can somehow ask OpenStack to live migrate OR if we cannot quiesce shutdown then snapshot 🤔
Edit: this isn't possible, we already have os_require_quiesce and if the GA isn't available it will still carry on to be "crash-safe`

Test cases that worked:

GPU -> shutdown gracefully, then restarted without intervention from myself or the "user"
Multiple migrations back and forth
Handles low space on source HV (1.5GB free, completed fine still)
Snapshots all created in ST2 service account's project
Concurrent migrations migrations work correctly
VM that has an error for a previous migration (e.g. no space on aggregate) re-tried and completed after making space
Qemu-GA offline migrates successfully with the caveat about snapshots above

We decide whether a Live Migration or a Cold Migration is performed on ACTIVE Servers based on the value of the new flag "live_migration". SHUTOFF Servers are still handled with a Cold Migration. Some refactoring to make the code more readable: * the step for both Live Migration and Cold Migration have been moved to their own functions * the check for Servers in a status different than ACTIVE and SHUTOFF has been moved outside the original can_be_migrated() function and moved into the new decision tree. This way, the can_be_migrated() function only checks the properties of the Server's Flavor.

DavidFair · 2026-06-30T16:04:41Z

@jose-caballero could you look at:

Attempting to live_migrate a GPU will snapshot first (could take hours) then will fail as it's blocked, need to check if the migration is permitted before doing any work

This should check for GPU or error before we do any steps - it's likely a bug in the existing implementation too

I'm currently looking at the endless wait if it fails:

If the source is disabled and the destination has no slots the ST2 action will sit pending despite it going into error on server migration list. I think there's a bug on detecting a migration (not server) going into error state and this caught me out several times and needs fixing before we do this for real

Noticed while testing cold migration. Openstack's wait for status does not allow us to specify invalid states. This means if we ask for it to go to "VERIFY_RESIZE" it will spend the entire timeout (previously 1 hour) waiting. Instead we should check the migration status for the error (as the VM will appear ACTIVE/SHUTDOWN the entire time unless there's a serious error), then confirm the VM moves to VERIFY_RESIZE on the happy path. Add some extra test cases in-case we ever have it a VM completes a migration but somehow ends up ACTIVE or ERROR when the migration is finished. This is extremely unlikely, but play defensive in this case.

DavidFair · 2026-06-30T18:58:02Z

Fixed the hang - need to be careful we both fetch before editing or rebasing from this point

This should check for GPU or VM's in invalid states/errors before we do any steps - it's likely a bug in the existing implementation too

I haven't made a start so if you beat me to it tomorrow feel free to push another commit onto this, then I think we're good to retest and maybe even

Skips any migration steps, as we would previously take a snapshot - force the user to wait for multiple hours then fail when we knew up-front it wouldn't work. Instead provide rapid feedback that we can't do an error state or we can't migrate before doing any work

DavidFair · 2026-07-01T09:39:29Z

Attempting to live_migrate a GPU will snapshot first (could take hours) then will fail as it's blocked, need to check if the migration is permitted before doing any work

#461 after this is reviewed + merged

If the source is disabled and the destination has no slots the ST2 action will sit pending despite it going into error on server migration list. I think there's a bug on detecting a migration (not server) going into error state and this caught me out several times and needs fixing before we do this for real

This now correctly works

2026-07-01 09:33:02,919 - apis.openstack_api.openstack_server - INFO - Migrating server: 4dbe774a-b036-4b1c-8323-ad167df31cbe
2026-07-01 09:33:03,363 - apis.openstack_api.openstack_server - INFO - Status of migration of server 4dbe774a-b036-4b1c-8323-ad167df31cbe: error
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.10/site-packages/python_runner/python_action_wrapper.py", line 397, in <module>
    obj.run()
---snipped----
    raise ResourceFailure(migration)
openstack.exceptions.ResourceFailure: openstack.compute.v2.migration.Migration(id=10603, uuid=528df2b2-d2c8-4fc8-b93e-89d7871fcc60, source_compute=hv-ref2023-a4000-26.nubes.rl.ac.uk, dest_compute=None, source_node=hv-ref2023-a4000-26.nubes.rl.ac.uk, dest_node=None, dest_host=None, old_instance_type_id=539, new_instance_type_id=539, instance_uuid=4dbe774a-b036-4b1c-8323-ad167df31cbe, status=error, migration_type=migration,
"
  stdout: ''

There's a separate bug where if the most recent migration goes into error it will re-migrate but incorrectly pick up on that previous error now we've got stricter checking. We'll need to do another PR for that but it's not a regression from this work since it would error before too....

And the migrations with enough space still work, so I'm going to mark it as tested on dev

    2026-07-01 09:38:19,244 - apis.openstack_api.openstack_server - INFO - Starting snapshot of server: 4dbe774a-b036-4b1c-8323-ad167df31cbe
    2026-07-01 09:38:53,886 - apis.openstack_api.openstack_server - INFO - Completed snapshot of server: 4dbe774a-b036-4b1c-8323-ad167df31cbe
    2026-07-01 09:39:03,896 - apis.openstack_api.openstack_server - INFO - Migrating server: 4dbe774a-b036-4b1c-8323-ad167df31cbe
    2026-07-01 09:39:04,416 - apis.openstack_api.openstack_server - INFO - Status of migration of server 4dbe774a-b036-4b1c-8323-ad167df31cbe: confirmed
    2026-07-01 09:39:09,443 - apis.openstack_api.openstack_server - INFO - Status of migration of server 4dbe774a-b036-4b1c-8323-ad167df31cbe: migrating
    2026-07-01 09:39:14,470 - apis.openstack_api.openstack_server - INFO - Status of migration of server 4dbe774a-b036-4b1c-8323-ad167df31cbe: post-migrating
    2026-07-01 09:39:19,498 - apis.openstack_api.openstack_server - INFO - Status of migration of server 4dbe774a-b036-4b1c-8323-ad167df31cbe: finished
    2026-07-01 09:39:19,601 - apis.openstack_api.openstack_server - INFO - Confirming resize for 4dbe774a-b036-4b1c-8323-ad167df31cbe
    2026-07-01 09:39:19,641 - apis.openstack_api.openstack_server - INFO - Migration completed of server: 4dbe774a-b036-4b1c-8323-ad167df31cbe
    '
  stdout: ''

DavidFair

Approved for Jose's work

@jose-caballero should review and approve mine with a comment (can't approve own PR) then we just need a second reviewer to merge

BUG: Skip any migration or snapshot steps if we're in an invalid state

…lid state"

…migration Revert "BUG: Skip any migration or snapshot steps if we're in an invalid state"

…lid state"

…migration Revert "BUG: Skip any migration or snapshot steps if we're in an invalid state"

DavidFair

Right take 2 - approved pending a second reviewer....

jose-caballero marked this pull request as draft June 29, 2026 09:01

jose-caballero force-pushed the add_flag_live_migration branch 3 times, most recently from 4597c50 to d76ebcc Compare June 29, 2026 12:19

jose-caballero force-pushed the add_flag_live_migration branch from f4ed702 to 2a84329 Compare June 29, 2026 15:14

DavidFair force-pushed the add_flag_live_migration branch from 2a84329 to 640c19b Compare June 30, 2026 14:49

DavidFair force-pushed the add_flag_live_migration branch 2 times, most recently from 0a7ddf7 to 48094bf Compare June 30, 2026 18:52

DavidFair force-pushed the add_flag_live_migration branch from 48094bf to 5e00a96 Compare June 30, 2026 18:57

jose-caballero marked this pull request as ready for review July 1, 2026 06:52

DavidFair mentioned this pull request Jul 1, 2026

BUG: Skip any migration or snapshot steps if we're in an invalid state #461

Merged

6 tasks

DavidFair previously approved these changes Jul 1, 2026

View reviewed changes

Merge pull request #461 from stfc/check_before_attempting_migration

dca5f0a

BUG: Skip any migration or snapshot steps if we're in an invalid state

jose-caballero dismissed DavidFair’s stale review via dca5f0a July 1, 2026 09:50

DavidFair added 4 commits July 1, 2026 10:51

Revert "BUG: Skip any migration or snapshot steps if we're in an inva…

5a173b1

…lid state"

Merge pull request #462 from stfc/revert-461-check_before_attempting_…

6863355

…migration Revert "BUG: Skip any migration or snapshot steps if we're in an invalid state"

Revert "BUG: Skip any migration or snapshot steps if we're in an inva…

039b7e1

…lid state"

Merge pull request #463 from stfc/revert-461-check_before_attempting_…

bdf2fa7

…migration Revert "BUG: Skip any migration or snapshot steps if we're in an invalid state"

DavidFair approved these changes Jul 1, 2026

View reviewed changes

anish-mudaraddi approved these changes Jul 1, 2026

View reviewed changes

anish-mudaraddi reviewed Jul 1, 2026

View reviewed changes

Comment thread lib/apis/openstack_api/openstack_server.py

jose-caballero merged commit 7b5f6db into main Jul 1, 2026
14 checks passed

jose-caballero deleted the add_flag_live_migration branch July 1, 2026 12:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add flag live migration#459

Add flag live migration#459
jose-caballero merged 9 commits into
mainfrom
add_flag_live_migration

jose-caballero commented Jun 29, 2026 •

edited by DavidFair

Loading

Uh oh!

codecov Bot commented Jun 29, 2026 •

edited

Loading

Uh oh!

DavidFair commented Jun 30, 2026 •

edited

Loading

Uh oh!

DavidFair commented Jun 30, 2026

Uh oh!

DavidFair commented Jun 30, 2026

Uh oh!

DavidFair commented Jul 1, 2026

Uh oh!

DavidFair left a comment

Uh oh!

DavidFair left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jose-caballero commented Jun 29, 2026 • edited by DavidFair Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Special Notes:

Submitter:

Reviewer

Uh oh!

codecov Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DavidFair commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidFair commented Jun 30, 2026

Uh oh!

DavidFair commented Jun 30, 2026

Uh oh!

DavidFair commented Jul 1, 2026

Uh oh!

DavidFair left a comment

Choose a reason for hiding this comment

Uh oh!

DavidFair left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jose-caballero commented Jun 29, 2026 •

edited by DavidFair

Loading

codecov Bot commented Jun 29, 2026 •

edited

Loading

DavidFair commented Jun 30, 2026 •

edited

Loading