Add Support for Migration Statistics API Call by phip1611 · Pull Request #43 · cyberus-technology/cloud-hypervisor

phip1611 · 2025-11-25T17:07:56Z

TL;DR

This extends the internal API with a vm_progress function and adds a vm.migration-progress HTTP endpoint including support in ch-remote via ch-remote migration-process to query the latest migration progress.

Motivation

Monitoring a live-migration with live updated information is very important for debugging, development, and monitoring. This is something that verbose logging can't achieve as there is a clear desire to have that structured information somewhere on the outside - and also to prevent spammy logs.

The most interesting part is the pre-copy phase where we get information on each new memory iteration. The first version is rather coarse-grained with one update per memory iteration. More to follow.

The ch driver in libvirt will use these information to populate its
virsh domjobinfo information.

Further, the endpoint will be interesting to query information about a
previously failed or canceled live migration.

Prerequisites

The two major pre-requisites were:

asynchronize the internal API vmm: live migration asynchronization (dedicated thread) #24
asynchronize the HTTP API Make HTTP Server multi threaded #52

Steps to Undraft

test it locally using ch-remote
add libvirt-tests testcase and verify everything works
finish refactoring for "non-blocking send-migration" (2026-02-03)
deploy it on a node in SAP land and see if it works
merge this when we know the libvirt part is also fine (I however think that this is 99% already)

vm-migration/src/progress.rs

src/bin/ch-remote.rs

vmm/src/api/mod.rs

vmm/src/lib.rs

phip1611 · 2026-01-13T07:40:53Z

vm-migration/src/progress.rs

+/// [live-migration protocol]: super::protocol
+#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
+pub struct MigrationProgressAndStatus {
+    /// UNIX timestamp of the start of the live-migration process.


please note that this structure will be public API for ever and once we reploy it, it will be hard to change

vm-migration/src/progress.rs

phip1611 · 2026-01-22T12:36:00Z

Thanks for your work on this!

I wish the PR description, or some other form of documentation, explained the intended use cases for this feature.

I've updated the code.

The metrics crate seems to be the metrics analogue of log/tracing and it would be good to know why that isn't a good fit for the task(s) addressed by this PR. I am assuming that there is some context I am missing and that it is quite possible that I have some misconceptions about the problems you are trying to address.

I don't see how this would help. These statistics are not "nice helpers that we need sometimes to debug". They are a fundamental and important functionality we want to export via API.

I hope the new PR description makes the intent here clearer.

amphi

Looks good, just a few small comments and nits.

vm-migration/src/progress.rs

amphi · 2026-01-22T14:27:13Z

vmm/src/lib.rs

+        // Give management software a chance to fetch the migration state.
+        // The VMM already executes on the other side and keeping Cloud Hypervisor running for a
+        // couple of more seconds is fine.
+        info!("Sleeping five seconds before shutting off.");


I think this waiting, and maybe also the mark_as_finished() a few lines above, should happen in check_migration_result. Otherwise the management software would see the mark_as_failed very late.

Agree about the sleep. Not sure about the other. Other thoughts?

Update. all of this need to happen in the end of the migration thread, not in check_migration_result. Otherwise, the API thread is blocked and API users can never query the latest state.

On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

The logging is not very spammy nor costly (iterations take seconds to dozens of minutes) and is clearly a win for us to debug things. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is the first commit in a series of commits to introduce a new API endpoint in Cloud Hypervisor to report progress and live-insights about an ongoing live migration. Having live and frequently refreshing statistics/metrics about an ongoing live migration is especially interesting for debugging and monitoring. For the first time, we will be able to see how live-migrations behave and create benchmarking infrastructure around it. The ch driver in libvirt will use these information to populate its `virsh domjobinfo` information. We will add a new API endpoint to query information. Further, the endpoint will be interesting to query information about a previously failed or canceled live migration. Specifically interesting about this API endpoint is that it will be the first endpoint that needs the "asynchronization" of the API: more than one API request in parallel. This needs support at least in the HTTP API and the internal API. The "SendMigration" call is long-running and active even if someone is querying the new endpoint. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. In this commit, we add the HTTP endpoint to export ongoing VM live-migration progress. This work was made possible because of the following fundamental prerequisites: - internal API was made async - http thread was made async This way, one can send requests to fetch the latest state without blocking in any code path of the API. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. This commit prepares the avoidance of naming clashes in the following. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. This commit actually brings all the functionality together. The first version has the limitation that we populate the latest snapshot once per memory iteration, although this is the most interesting part by far. In a follow-up, we can make this more fine-grained. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live-migration. See the first commit for an introduction. There isn't really an error that can happen when we query this endpoint. A previous snapshot may either be there or not. It also doesn't make sense here to check if the current VM is running, as users should always be able to query information about the past (failed or canceled) live migration. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

phip1611 self-assigned this Nov 25, 2025

phip1611 force-pushed the poc-migration-statistics branch from e29356d to 4101b11 Compare November 28, 2025 08:46

phip1611 commented Nov 28, 2025

View reviewed changes

vm-migration/src/progress.rs Show resolved Hide resolved

tpressure reviewed Nov 28, 2025

View reviewed changes

vm-migration/src/progress.rs Outdated Show resolved Hide resolved

phip1611 force-pushed the poc-migration-statistics branch 5 times, most recently from f49e577 to fdf5858 Compare December 9, 2025 11:42

phip1611 commented Dec 9, 2025

View reviewed changes

src/bin/ch-remote.rs Outdated Show resolved Hide resolved

phip1611 commented Dec 9, 2025

View reviewed changes

vmm/src/api/mod.rs Outdated Show resolved Hide resolved

phip1611 force-pushed the poc-migration-statistics branch from f612cf6 to e9a3321 Compare December 15, 2025 14:24

phip1611 force-pushed the poc-migration-statistics branch from e9a3321 to ecb5f45 Compare January 8, 2026 14:31

phip1611 changed the base branch from gardenlinux-v48 to gardenlinux January 8, 2026 14:32

phip1611 force-pushed the poc-migration-statistics branch 3 times, most recently from e6c80dd to fe8cd0d Compare January 12, 2026 16:30

phip1611 marked this pull request as ready for review January 12, 2026 16:30

phip1611 changed the title ~~WIP XXX Migration Statistics~~ Add Support for Migration Statistics API Call Jan 12, 2026

phip1611 requested review from amphi and olivereanderson January 12, 2026 16:33

phip1611 commented Jan 12, 2026

View reviewed changes

vmm/src/lib.rs Outdated Show resolved Hide resolved

phip1611 commented Jan 12, 2026

View reviewed changes

vmm/src/lib.rs Outdated Show resolved Hide resolved

phip1611 marked this pull request as draft January 12, 2026 16:35

phip1611 force-pushed the poc-migration-statistics branch from fe8cd0d to a87fe95 Compare January 12, 2026 16:41

phip1611 requested a review from tpressure January 12, 2026 16:42

phip1611 force-pushed the poc-migration-statistics branch from a87fe95 to 6d68c0c Compare January 12, 2026 16:43

phip1611 commented Jan 13, 2026

View reviewed changes

vm-migration/src/progress.rs Outdated Show resolved Hide resolved

phip1611 force-pushed the poc-migration-statistics branch from 6d68c0c to c934e0e Compare January 13, 2026 10:36

phip1611 force-pushed the poc-migration-statistics branch from 708695d to 40e58e3 Compare January 22, 2026 12:31

phip1611 requested a review from olivereanderson January 22, 2026 12:31

phip1611 force-pushed the poc-migration-statistics branch 3 times, most recently from ceb915e to 5b63ea5 Compare January 22, 2026 14:27

amphi reviewed Jan 22, 2026

View reviewed changes

phip1611 force-pushed the poc-migration-statistics branch 3 times, most recently from a83f554 to 7ee8334 Compare January 27, 2026 14:50

phip1611 mentioned this pull request Jan 28, 2026

vmm: migration: streamline and cleanup memory_copy_iterations() #72

Merged

1 task

phip1611 force-pushed the poc-migration-statistics branch from 7ee8334 to f937d87 Compare January 29, 2026 12:10

phip1611 requested review from amphi and scholzp January 29, 2026 12:10

phip1611 force-pushed the poc-migration-statistics branch 2 times, most recently from 679c6f1 to 307475e Compare February 2, 2026 13:03

phip1611 marked this pull request as draft February 3, 2026 16:20

phip1611 force-pushed the poc-migration-statistics branch from 307475e to f32b6f7 Compare February 3, 2026 16:24

phip1611 added 11 commits February 4, 2026 20:08

vmm: silence some messages in http_endpoint

9e3c2ed

On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

xxx

e632efe

xxx

6b18d8f

phip1611 force-pushed the poc-migration-statistics branch from b171d24 to 6b18d8f Compare February 4, 2026 19:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Support for Migration Statistics API Call#43

Add Support for Migration Statistics API Call#43
phip1611 wants to merge 11 commits intocyberus-technology:gardenlinuxfrom
phip1611:poc-migration-statistics

phip1611 commented Nov 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phip1611 Jan 13, 2026

Uh oh!

Uh oh!

phip1611 commented Jan 22, 2026

Uh oh!

amphi left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amphi Jan 22, 2026

Uh oh!

phip1611 Jan 22, 2026

Uh oh!

phip1611 Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

phip1611 commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TL;DR

Motivation

Prerequisites

Steps to Undraft

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phip1611 Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

phip1611 commented Jan 22, 2026

Uh oh!

amphi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amphi Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

phip1611 Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

phip1611 Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

phip1611 commented Nov 25, 2025 •

edited

Loading