kubevirt-vm-migration-platform

Stage 0: theory, architecture and feasibility

Project 1.5: VM-to-Kubernetes Migration Platform (ESXi -> KubeVirt Migration & VM Modernization)

Stage 0 status: complete, with a technical refinement pass (2026-09-27), pending user review. Nothing was installed or created. ESXi, VMware networking, Windows virtualization and AWS are unchanged.

Later outcome (note added 2026-10-15). This page records Stage 0 as it stood on 2026-09-27. Stage 1 was then approved and carried out in sub-stages 1A to 1I. Stage 1I completed the first cold migration from ESXi to KubeVirt on a temporary, Terraform-built, single-node kubeadm cluster on AWS EC2, validated it, and destroyed the target (2026-10-01). Several “unvalidated” items below were proven along the way; see the status table added to the feasibility page and the Stage 1 index. The S3 staging step in the one-line architecture was not used: Stage 1I transferred the image over SFTP and used a CDI upload (ADR 009). The migration controller (ADR 005) and the containerized version remain unimplemented.

Why Stage 0 exists

Migration tooling hides a long chain of layers: VMware objects, disk formats, guest drivers, KVM/QEMU/libvirt, KubeVirt objects, CDI, Kubernetes storage and networking, and controller design. If we start installing before we understand that chain, every failure becomes guesswork. Stage 0 builds the mental model first, checks current (not remembered) version facts, and identifies what must be proven before any build.

Documents

Document Covers
virtualization-fundamentals.md ESXi VM vs KubeVirt VM vs Pod; the ESXi -> guest OS chain and owners; KVM, QEMU, libvirt, VirtIO; VMware -> KVM translation table; nested virtualization (L0/L1/L2, VT-x/EPT)
kubevirt-architecture.md Current KubeVirt components, VM vs VMI, lifecycle from kubectl apply to guest boot, install requirements
crd-controller-fundamentals.md CRDs, controllers, reconcile, status, conditions, finalizers; why a controller instead of scripts; the controller model mapped to our state machine
cdi-storage-model.md CDI, DataVolume, PVC, import/upload/clone; where CDI ends and KubeVirt begins; VMDK/QCOW2/RAW; qemu-img, virt-v2v, libguestfs; storage mapping
vmware-source-model.md Our ESXi objects, what to discover, the inventory model, metadata vs disk data vs runtime state
migration-architecture.md End-to-end migration flow, cold vs warm, MTV/Forklift as reference vs our platform, the conceptual VirtualMachineMigration CRD, the first demo design
migration-state-machine.md Phases, automatic / retryable / terminal transitions, idempotency
networking-model.md Pod network, masquerade, bridge, Multus, NAD, secondary networks vs vSwitch / port group / vNIC; network mapping
modernization-model.md VM migration vs application modernization, worked nginx example
architecture.md End-to-end architecture, component responsibilities, lab resource architecture (local + disposable AWS)
feasibility.md What must be validated before Stage 1, with status and sources
evidence-index.md Summarized, non-sensitive evidence of the current lab
glossary.md Terms, with VMware analogies
interview-notes.md 13 interview questions with short and detailed answers

Diagrams (all in ../diagrams/):

Diagram Used in
virtualization-stack-comparison.svg virtualization-fundamentals
kubevirt-object-chain.svg virtualization-fundamentals, architecture
nested-virtualization-layers.svg virtualization-fundamentals
kubevirt-architecture.svg kubevirt-architecture
cdi-import-flow.svg cdi-storage-model
migration-flow.svg migration-architecture
migration-state-machine.svg migration-state-machine
controller-reconcile-loop.svg crd-controller-fundamentals
networking-model.svg networking-model
lab-resource-architecture.svg architecture
modernization-paths.svg modernization-model

ADRs: ../adr/README.md.

What was learned

  1. A KubeVirt VM is still a VM. The guest OS, kernel and disk are the same as on ESXi. What changes is the manager: VMkernel/vCenter becomes Linux KVM + QEMU + libvirt, driven by Kubernetes objects.
  2. The chain and its owners: VirtualMachine (you / controller) -> VirtualMachineInstance (virt-controller) -> virt-launcher Pod (scheduler + kubelet) -> libvirt domain (virt-handler) -> QEMU (libvirt) -> KVM (node kernel) -> guest OS (app team).
  3. CDI ends at the PVC. CDI turns an image into a raw disk in a PVC. KubeVirt starts when a VM references that PVC.
  4. Format conversion is not guest conversion. qemu-img changes the container format; virt-v2v also fixes drivers and boot config for virtio.
  5. Metadata is translated, disk data is copied, runtime state is only observed.
  6. Cold first. Warm migration depends on snapshots and CBT via the vSphere API, which the free ESXi edition does not support.
  7. A controller, not scripts, is the right owner of a long, failure-prone, multi-step workflow. The state machine is a set of declarative reconciliation checkpoints, not a procedural script; idempotency, resumability and retry safety are the core design rules.
  8. A VMDK’s file layout depends on its format and provisioning mode (descriptor + extents, or monolithic sparse); discovery must read it, not assume it.
  9. Versions are a tuple, validated at deployment time. “Latest at documentation time” is an observation, not a decision.
  10. Migration preserves the VM model; modernization removes it.

What was validated (in our lab)

Only facts we observed ourselves. Full list: feasibility.md, “Proven in our lab”.

Item How
Nested ESXi 8.0.3 build 24677879 runs on VMware Workstation Read-only SSH (see evidence-index.md)
ESXi sees hardware virtualization (HV Support: 3) Read-only SSH
VMFS-6 migration-datastore 199.8 GB / 198.3 GB free, no VMs Read-only SSH
Static vmk0 192.168.50.11, SSH key auth, NTP in sync Read-only SSH

What was confirmed from documentation (documented, NOT validated)

Item Source
KubeVirt v1.9 is built for Kubernetes 1.36 and supported on the previous two minors KubeVirt release notes / support matrix
Kubernetes 1.34 EOL 2026-10-27; EKS standard support 1.34-1.36 kubernetes.io, AWS docs
CDI v1.66.1 is the newest release; no official KubeVirt-to-CDI pairing statement found CDI releases / API reference
AWS nested virtualization on supported non-bare-metal instances since 2026-02-16; KVM and Hyper-V; no additional cost AWS EC2 user guide
Free ESXi: API unsupported / possibly read-only, no vCenter, 8 vCPU per VM Broadcom KB + release notes
virt-v2v -i vmx -it ssh requirements (guest shut down, no snapshots, key auth) virt-v2v-input-vmware(1)
MTV 2.11 architecture and warm-migration limits Red Hat docs

What remains unvalidated

See feasibility.md, “Not yet proven”. Most important:

Architecture

See architecture.md. In one line: local nested ESXi source -> SSH discovery and cold disk copy -> conversion -> S3 staging -> CDI DataVolume/PVC on a disposable Terraform-built AWS Kubernetes cluster -> KubeVirt VM -> validation -> containerized version of the same app.

Decisions

Decision Record
Local nested ESXi is the source ADR 001
AWS is the disposable target, Terraform only ADR 002
Cold before warm; no custom warm engine ADR 003
Upstream KubeVirt + CDI is the destination; versions chosen as a validated compatibility tuple at deployment time ADR 004
Custom controller + VirtualMachineMigration CR owns the declarative migration workflow (reconciliation checkpoints) ADR 005

Deferred decisions:

Stage 1 entry criteria

Stage 1 may begin only when all general criteria are met, plus the criteria for whichever track Stage 1 covers.

General (always required):

  1. The user has reviewed Stage 0 (including this refinement) and explicitly approved starting Stage 1.
  2. Stage 1 scope is written down: which track (source side, destination side, or both), what will be created, and what is out of scope.
  3. The approved change boundary is explicit: which systems Stage 1 may modify (ESXi, AWS, Kubernetes). Anything not listed stays unchanged.
  4. Deferred decisions needed by that scope are either made (recorded as an ADR) or explicitly kept out of scope.
  5. Success criteria and evidence to capture are defined for each Stage 1 task, using the categories in feasibility.md.

Source-side track (creating legacy-source-vm on ESXi):

  1. A controlled ESXi reboot has been approved, performed, and vmk0/route/DNS/hostname/NTP/datastore/SSH re-verified (N2).
  2. Guest OS, version, ISO (with checksum) and sizing (at most 8 vCPU) are chosen.
  3. The ISO upload location on migration-datastore and the VM’s port group (“VM Network”) are agreed.
  4. The first Stage 1 check is a nested 64-bit guest boot (N1). If it fails, stop and report.

Destination-side track (any AWS or Kubernetes work):

  1. AWS credentials are recovered from the Docker volume or re-issued, stored outside Git, and verified with a read-only call (F20).
  2. Terraform is the only creation path; a terraform destroy plan and cost guardrails (tags, budget alert, session teardown) are defined.
  3. A candidate compatibility tuple (KubeVirt, Kubernetes, CDI) is written down with its evidence, and the runtime validation gate G1-G7 is the acceptance test (feasibility section 3).
  4. The first test node is scoped to the host-requirement checklist H1-H11 (feasibility section 5), for whichever topology is tried first; EKS vs self-managed is decided from that evidence.
  5. Instance family, size, Region and AMI for the test node are chosen and recorded (currently deferred).

Migration track (only after both sides exist):

  1. The conversion host location is decided and recorded as an ADR.
  2. The source VM exists, runs nginx, has no snapshots, and has a recorded baseline (page content, checksums).

Official sources consulted

All checked on 2026-09-27.

KubeVirt:

CDI:

Kubernetes:

Red Hat MTV / Forklift:

libvirt, QEMU, KVM, libguestfs:

VMware / Broadcom:

AWS:

Multus: