kubevirt-vm-migration-platform

Migration state machine

Stage 0 design document for request section 13. It defines the status.phase values of the conceptual VirtualMachineMigration CR (see migration-architecture.md) and how a controller moves between them (see crd-controller-fundamentals.md). Not implemented.

Migration state machine

1. Reconciliation checkpoints, not a procedural script

The migration state machine represents declarative reconciliation checkpoints, not a procedural script.

A script says “do step 5, then step 6”. A controller says “this is what should be true; let me look at what is true; let me do whatever closes the gap; let me record what I saw; let me look again”. Each phase is therefore a checkpoint that names the desired state the controller is currently working to make true.

status.phase is not merely a log of what happened. It is the controller’s current reconciliation state: the checkpoint it is working on right now. On every reconcile, including after a controller crash or restart, the controller reads the phase, re-observes the real system, and decides again. History belongs in status.errors, status.timestamps, conditions and Kubernetes Events, not in the phase.

Terms

Term Meaning in this design
Desired state What must be true to leave the current checkpoint (for example “a valid converted artifact exists”). Derived from spec + the current phase.
Observed state What the controller sees now when it looks at the real system (files, objects, DataVolume/VMI status, ESXi power state). Never assumed from memory.
Idempotency Running the same reconcile step twice has the same effect as running it once.
Resumability After any interruption (crash, restart, network loss), the next reconcile continues from the observed state instead of starting over.
Retry safety A failed step can be retried without corrupting data or duplicating objects, because partial results are detected and either resumed or cleaned up first.

Example: the Converting checkpoint

Not this (procedural):

Converting
   |
   v
run virt-v2v once

But this (declarative checkpoint):

Desired state:     a converted artifact for every disk exists, is valid (checksum/qemu-img check),
                   and is staged where CDI can reach it

Observed state:    one of
                     absent    - no conversion Job/output for this disk
                     partial   - Job running, or output present but incomplete / checksum missing
                     complete  - output present, validated, staged, checksum recorded
                     failed    - Job failed, or output invalid

Controller action: absent   -> start conversion (Job with a deterministic name)
                   partial  -> resume: wait for the running Job, or remove the invalid partial output and restart
                   complete -> nothing to do
                   failed   -> classify: transient -> Retryable; permanent -> Failed / ManualInterventionRequired

Status:            update phase, conditions (DiskConverted), progress (bytes/percent), error information

Next reconcile:    re-observe the system; decide whether conversion is complete;
                   continue (advance to Importing), retry, or fail

The same model for every active checkpoint

Checkpoint Desired state Observed states Controller action Status written Next reconcile
Preparing Source VM is powered off (per powerOffPolicy) and every disk has been acquired intact Power: on / shutting down / off. Disk copy: absent / partial / complete (size + checksum match) / failed Request graceful shutdown only if on and allowed; wait if shutting down; copy only when observed off; resume or restart a partial copy SourcePoweredOff, DiskAcquired, timestamps.sourcePoweredOff (set once), copy progress, errors Re-check power state and copy result; advance to Converting, retry, or ManualInterventionRequired (for example guest will not shut down)
Converting Valid converted artifact staged per disk absent / partial / complete / failed (see above) Start / resume / clean up + restart / classify failure DiskConverted, progress, artifact URL + checksum, errors Advance to Importing, retry, or fail
Importing One DataVolume per disk has Succeeded DataVolume absent / Pending or WaitForFirstConsumer / ImportInProgress / Succeeded / Failed Create if absent (deterministic name, owner reference); wait if in progress; on transient failure delete and recreate within budget DiskImported, status.progress copied from DataVolume, DataVolume names, errors Advance to CreatingVM, keep waiting, retry, or fail
CreatingVM A VirtualMachine owned by this migration exists and matches the rendered spec absent / present and ours but drifted / present and matching / present but not ours Create if absent; update if ours and drifted; nothing if matching; stop if not ours VMCreated, destinationVM (name, UID), errors Advance to Starting, or ManualInterventionRequired (name conflict)
Starting The VMI is Running runStrategy not yet set / VMI absent / Pending or Scheduling / Running / Failed / timeout exceeded Set runStrategy (declarative, safe to re-apply); wait; on scheduling failure classify (capacity -> Retryable; missing /dev/kvm -> ManualInterventionRequired) VMStarted, timestamps.vmStarted, VMI phase, errors Advance to Validating, keep waiting, retry, or intervene
Validating Every spec.validation.checks entry passes not yet run / some failing / all passing / timed out Run read-only checks (HTTP via Service, guest agent, IP); repeat until pass or timeout GuestReady, per-check results, errors Advance to Completed, keep checking, or ManualInterventionRequired (app broken)

2. Phases

Pending
  -> Discovering
  -> Validated
  -> Preparing
  -> Converting
  -> Importing
  -> CreatingVM
  -> Starting
  -> Validating
  -> Completed

Failure states: Retryable, ManualInterventionRequired, Failed
Phase (checkpoint) Desired state (what must be true to leave it) Controller action (idempotent, re-entrant) Observed state that advances it
Pending The CR is accepted and has a finalizer. Add finalizer; set timestamps.created. Finalizer present.
Discovering Source VM found and inventory recorded. SSH to ESXi, vim-cmd + read .vmx. Read-only. status.sourceInventory written; SourceDiscovered=True.
Validated Pre-flight checks pass. (Named for its result: while in this phase the controller runs the checks.) Evaluate rules: guest OS, firmware, no snapshots, disk sizes vs quota, StorageClass exists, network mapping covers every NIC, destination name free. Validated=True, or blockers recorded.
Preparing Source is quiesced and the disk is acquired. Power off per powerOffPolicy; confirm poweredOff; copy VMDK; record checksum. SourcePoweredOff=True, DiskAcquired=True.
Converting Converted image exists and is staged. qemu-img / virt-v2v; upload to staging; record URL + checksum. DiskConverted=True.
Importing Every disk is imported into a PVC. Ensure DataVolume per disk (deterministic names). All DataVolumes Succeeded; DiskImported=True.
CreatingVM The VirtualMachine exists with the right spec. Create-or-update VirtualMachine (initially runStrategy: Halted). VM exists and matches the rendered spec; VMCreated=True.
Starting The VMI is running. Set runStrategy to the requested value (for example Always). VMI status.phase == Running within the timeout.
Validating The application works. Run spec.validation.checks (HTTP probe through a Service, guest agent, IP). All checks pass; GuestReady=True.
Completed Terminal success. Clean staging data; set Ready=True, timestamps.completed. -
Retryable A step failed in a way that may succeed on retry. Wait with backoff; undo the partial step if needed; go back to the failed phase. Retry started, or budget exhausted -> Failed.
ManualInterventionRequired A human decision or fix is needed. Nothing automatic. Record reason; emit Event. A human changes spec or adds an annotation (for example migration.platform.example/retry: "true").
Failed Terminal failure. Clean up destination artifacts per policy; never touch the source disk. -

3. Transition types

Automatic (controller advances on observed success)

Pending -> Discovering -> Validated -> Preparing -> Converting -> Importing -> CreatingVM -> Starting -> Validating -> Completed

Each arrow is taken only after the controller has observed the phase’s goal, not merely after it issued a command.

Retryable (go to Retryable, then back to the same phase)

From Example causes
Discovering SSH timeout, ESXi busy
Preparing Copy interrupted, checksum mismatch, transient disk-full on the conversion host
Converting Conversion process killed, staging upload failed
Importing DataVolume Failed with a transient reason (network, pod eviction)
CreatingVM API conflict (optimistic concurrency), webhook timeout
Starting VMI failed to schedule because of temporary capacity
Validating Application not yet ready within the first timeout

Rules: bounded by spec.strategy.retryLimit, exponential backoff, the attempt is recorded in status.errors, and the retry resumes the same phase, never restarts from Pending.

Manual intervention (go to ManualInterventionRequired)

From Example causes
Validated Source VM has snapshots; unsupported firmware; a NIC has no network mapping
Preparing Guest will not shut down gracefully and policy forbids a hard power-off
Starting Guest boots but kernel cannot find root disk (driver problem)
Validating App reachable but returns wrong content
Retryable Retry budget exhausted for a step that a human may fix (optionally, instead of Failed)

Terminal

Terminal states are never left automatically. Re-running means creating a new CR (or an explicit, documented reset annotation).

4. Where idempotency is required

Everywhere a phase can be re-entered. Because reconcile can run many times for the same phase (requeues, watch events, controller restarts), every external action must be safe to repeat.

Phase What could be duplicated Idempotency technique
Discovering None (read-only) Overwrite inventory; compare with the previous one to detect drift.
Preparing: power off Second power-off on an already-off VM Check power.getstate first; treat “already off” as success. Record timestamps.sourcePoweredOff once.
Preparing: disk copy Partial or duplicate copies Deterministic target path; compare size + checksum; resume or overwrite, never append.
Converting Double conversion, duplicate uploads Deterministic object key (<migration-uid>/disk-0.qcow2); skip if an object with the recorded checksum exists.
Importing Two DataVolumes for one disk Deterministic name <name>-disk-<n>; create-if-absent; owner reference to the migration CR.
CreatingVM Two VMs, or overwriting someone else’s VM Deterministic name; if it exists, check the owner reference / label. Ours: update to match. Not ours: ManualInterventionRequired.
Starting Repeated starts Setting runStrategy is declarative; re-applying the same value is a no-op.
Validating Repeated probes Read-only checks.
Completed / cleanup Deleting twice Treat “not found” as success.

Two more rules:

  1. Record before you advance. Write evidence (checksum, URL, object name) and the new phase in the same status update. If the update fails, the next reconcile repeats the step, which is safe because the step is idempotent.
  2. Irreversible actions need a guard. Powering off the source is the only action that affects the running service. It happens only after Validated=True, only per powerOffPolicy, and it is recorded with a timestamp so it is never “re-decided”.

5. Failure halfway: what the state machine guarantees