Project 1.5: VM-to-Kubernetes Migration Platform (ESXi -> KubeVirt Migration & VM Modernization)
Stage 1 turns the Stage 0 design into a working source environment and, in Stage 1I, the first end-to-end cold migration to KubeVirt, one small, separately approved step at a time. Each sub-stage has its own scope, change boundary, evidence and entry criteria for the next step.
Current status (2026-10-01): all sub-stages 1A to 1I are complete. Stage 1I migrated legacy-source-vm to KubeVirt on a temporary single-node kubeadm cluster on AWS EC2, validated it end to end (HTTP 200, 267 bytes, source page hash), and then destroyed the target and verified the teardown. No stage after 1I is defined.
Background: Stage 0 README, Stage 0 feasibility, ADRs.
| Sub-stage | Objective | Status | Document |
|---|---|---|---|
| 1A | Prove the ESXi source host survives a controlled reboot and all required configuration persists | Complete (PASS), 2026-09-27 | stage-1a-esxi-reboot-validation.md |
| 1B | One Linux learning VM on ESXi: KVM / QEMU / libvirt / VirtIO fundamentals and nested KVM feasibility | Complete (PASS), 2026-09-27 | stage-1b-kvm-qemu-fundamentals.md, kvm-learning-lab.md, nested-kvm-feasibility.md |
| 1C | Create and baseline the VMware migration source VM legacy-source-vm (Ubuntu 24.04.5, nginx, PVSCSI, VMXNET3, static IP) |
Complete (PASS), 2026-09-27 | stage-1c-migration-source.md |
| 1D | Characterize the source VM’s datastore artifacts and VMDK; investigate free-ESXi acquisition; design the cold copy (not executed) | Complete (PASS), 2026-09-27 | stage-1d-vmware-source-artifacts.md |
| 1E | Approved cold acquisition of the source disk (graceful shutdown, scp of descriptor + flat extent, sha256 verification, power-on and validation) |
Complete (PASS), 2026-09-27 | stage-1e-cold-acquisition.md |
| 1F | Dedicated conversion host conversion-host-01 with qemu-img, libguestfs and virt-v2v; hash-verified working copy of the acquired disk; read-only inspection (no conversion) |
Complete (PASS), 2026-09-27 | stage-1f-conversion-host.md |
| 1G | Controlled conversion experiments (qemu-img and virt-v2v) on disposable working data, QEMU/KVM boot validation with UEFI and virtio, minimal network remediation on a disposable copy; ADR 006 | Complete (PASS), 2026-09-28 | stage-1g-controlled-conversion.md, ADR 006 |
| 1H | KubeVirt destination feasibility and network/storage architecture (design only: nothing provisioned or installed); ADRs 007 to 010 | Done (design): H1 to H10 PASS, 2026-09-28; architecture review corrections applied 2026-09-30 (review). The design was then implemented in Stage 1I | stage-1h-kubevirt-target-feasibility.md, ADR 007, ADR 008, ADR 009, ADR 010 |
| 1I | First KubeVirt migration: prepared image (I1), pre-transfer boot test (I2), AWS destination platform (I3), first migration to KubeVirt (I4), final validation and controlled teardown (I5) | Complete (PASS): I1 to I5 PASS; completed 2026-10-01. The temporary AWS target was destroyed and verified in Gate I5 | stage-1i-first-kubevirt-migration.md, stage-1i-runtime-snapshot.md, stage-1i-visual-learning.html |
Diagrams: lab-nested-kvm-layers.svg (lab layers, Stage 1B), stage-1c-source-lab.svg (VM roles after Stage 1C), stage-1d-source-artifacts.svg (source VM artifacts and the future conversion boundary), stage-1f-conversion-toolchain.svg (golden artifact, working copy and the inspection toolchain), stage-1g-conversion-and-boot.svg (the two conversion paths and the QEMU/KVM boot tests), and the Stage 1H designs stage-1h-target-platform.svg (chosen KubeVirt target), stage-1h-migration-data-path.svg (VMDK to running VM) and stage-1h-network-model.svg (masquerade network path and IP continuity). The Stage 1I set in stage-1i-visual/ shows what was actually built and observed, for example the pipeline, infrastructure, packet path, storage lifecycle and evidence dashboard.
Labels used in all Stage 1 documents: OBSERVED = seen in this lab by a command we ran. INFERRED = reasoned from observations or documentation, not directly tested. NOT TESTED = deliberately not done.
A graceful esxcli system shutdown reboot was performed with the host in maintenance mode and no guest VMs. The host was back on the management network 40 s later (host log) and every checked item persisted: hostname, ESXi 8.0.3 build 24677879, vmk0 192.168.50.11/24 static, default route and DNS 192.168.50.2, vSwitch0 with vmnic0 and both port groups, migration-datastore (VMFS-6, byte-identical free space), HV Support 3, hostd, rhttpproxy, vpxa, SSH and ESXi Shell, HTTPS 443 and SSH key login. NTP re-synchronized on its own within about 3.5 minutes. Maintenance mode was then exited. This closes Stage 0 item N2 (reboot persistence).
One Ubuntu 24.04.4 LTS VM, kvm-learning-01 (2 vCPU, 4 GB, 40 GB thin, one VMXNET3 NIC on VM Network, UEFI, vhv.enable = TRUE), was installed unattended on migration-datastore. It booted as a 64-bit guest, which closes Stage 0 item N1. Inside it, ESXi exposes VT-x and EPT: /dev/kvm exists, and kvm_intel loads with nested=Y. With a minimal package set (QEMU 8.2.2, libvirt 10.0.0, cpu-checker), a tiny L3 guest booted KVM accelerated, both directly and through libvirt, and showed virtio-blk, virtio-scsi and virtio-net devices. CPU-bound work in L3 runs at near-L2 speed. VM exits are very expensive three hypervisors deep, so this lab is for learning and functional tests, not performance numbers. Details: Stage 1B record.
The migration source VM legacy-source-vm (Vmid 2: 2 vCPU, 4096 MB, 40 GB thin, UEFI, PVSCSI, one VMXNET3 NIC on VM Network) was installed unattended from the official Ubuntu 24.04.5 live-server ISO. The ISO was downloaded from releases.ubuntu.com, and its SHA256 and signature were verified. The VM serves a deterministic nginx page (267 bytes, fixed sha256) at the static address 192.168.50.31, reachable from Windows and ESXi. The baseline records the stable identifiers (filesystem and partition UUIDs, machine-id, SSH host keys, config checksums) and the VMware-specific ones that will change on KubeVirt: disk and NIC drivers, device names, MAC, SMBIOS UUID and UEFI NVRAM. The biggest recorded risk is that netplan matches the NIC by the name ens192, so a virtio-net NIC would come up unconfigured. As required, nothing was adapted. kvm-learning-01 was then shut down gracefully. The conversion host remains deferred. Details: Stage 1C record.
A read-only investigation of legacy-source-vm on the datastore. The disk is an ESXi vmfs VMDK: a 541-byte text descriptor plus one thin flat extent (40 GiB logical, 3,487 MiB allocated), with no snapshot chain. The VM configuration (.vmx) and the UEFI variable store (.nvram) are separate files, and neither is part of what a migration moves: only the disk bytes travel. While the VM runs, the flat extent is exclusively locked and cannot be read at all, so acquisition must be cold. On this free ESXi host, SSH/scp, vmkfstools -i (including a lossless 2gbsparse export, proven on a throwaway scratch disk) and the authenticated HTTPS datastore file service work. The recommended first acquisition is a cold scp of descriptor plus flat extent, with sha256 on both ends: about 40 GiB, 15 to 20 minutes of downtime. It is designed, not executed. The source was verified unchanged before and after. Details: Stage 1D record.
With explicit approval, legacy-source-vm was shut down gracefully through VMware Tools: an OS halt taking about 3 s, cleanShutdown = TRUE. It was confirmed off with its disk lock released. Its VMDK descriptor and 40 GiB flat extent were then copied with scp to C:\VMs\legacy-source-vm\stage-1e\, outside Git, in 276 s. The flat extent hashed to the same sha256 (72ca45c7...c9e7) on the source before the copy, on the copy, and on the source after the copy, so the copy is byte-identical and the source was not altered. The VM was powered back on, and networking, nginx and every baseline identifier matched. Service downtime was about 10.5 minutes. One Stage 1D assumption was disproved: the descriptor CID does not track writes on this disk type, and the Stage 1D record was corrected. Details: Stage 1E record.
With explicit approval, a dedicated Ubuntu 24.04.5 VM, conversion-host-01 (Vmid 3: 4 vCPU, 8 GB, 60 GB thin, UEFI, PVSCSI, VMXNET3, 192.168.50.32 after proving the address free), was installed unattended from the re-verified Stage 1C ISO. A minimal, investigated package set (--no-install-recommends) provides qemu-img 8.2.2, libguestfs/guestfish 1.52.0 and virt-v2v 2.4.0. KVM works inside it (vhv.enable), and the libguestfs appliance really runs under KVM, but three hypervisors deep TCG boots the appliance faster. The golden Stage 1E artifact was only read: scp made a working copy in /srv/migration-lab/working/, whose sha256 matched the golden before and after re-sparsifying, and which was then locked read-only and immutable. qemu-img, partition probing, libguestfs and guestfish --ro read it at the container, partition and guest-filesystem levels, and the results match the Stage 1C baseline. Migration-sensitive state was listed, not fixed: netplan keyed on ens192, cloud-init disabled, open-vm-tools enabled, virtio built into the kernel, UEFI fallback loader present. virt-v2v was studied from its own documentation only. Nothing was converted, and the golden artifact and source VM are unchanged. Details: Stage 1F record.
With explicit approval, the conversion-host decision was recorded as ADR 006, and the golden artifact was re-verified and given the Windows read-only attribute (content unchanged). Two conversions ran from the protected working copy into new directories. Path A, qemu-img convert -f vmdk -O qcow2, took 5 s and changed only the container: qemu-img compare reports the images identical. Path B, virt-v2v 2.4.0 -i disk -if vmdk -o local -of qcow2, took 136 s and left its input unchanged. It purged open-vm-tools, rebuilt the initramfs, added a virtio alias and a first-boot job that installs qemu-guest-agent, and wrote libvirt XML (q35, OVMF, virtio). It did not touch netplan, fstab, GRUB configuration or identity. Both disks booted under plain QEMU/KVM with OVMF (Secure Boot off, fresh varstore, ESP fallback loader), virtio-blk (vda) and virtio-net. In both, the NIC came up as enp0s3 while netplan still matched ens192, so the guest was healthy but had no network. A second copy of the virt-v2v output with one netplan change (match by driver virtio_net, test address 10.0.2.15 on an isolated network, never .31) came up routable, and nginx answered HTTP 200 with the source page hash from outside the guest. The golden artifact, the working copy and legacy-source-vm are unchanged. Details: Stage 1G record.
With explicit approval for design work only, the KubeVirt destination was chosen from official documentation, release artifacts and KubeVirt’s own CI configuration; no AWS resource, cluster or Kubernetes object was created. KubeVirt v1.9.0 (current) supports Kubernetes 1.36, 1.35 and 1.34, and its virt-launcher userland is built on CentOS Stream 9, whose kernel it recommends matching. ADR 007 chooses a self-managed, single-node kubeadm cluster on one m8i.xlarge EC2 instance with nested virtualization in ap-south-1: CentOS Stream 9, CRI-O 1.36, Kubernetes 1.36, flannel, the EBS CSI driver, KubeVirt v1.9.0 and CDI v1.66.1 (the CDI pairing is a CI-evidenced candidate, confirmed only at runtime). EKS was investigated and deferred: it is not blocked, but its node images are not EL-based and it adds a control-plane charge. The design costs about USD 0.24 per hour while running. ADR 008 uses the pod network with the masquerade binding and no Multus: the guest gets 10.0.2.2 by DHCP, and a NodePort Service reached only from the operator’s /32 is the external HTTP check. 192.168.50.31 is not preserved, and service continuity is proven by content, not by address. ADR 010 makes the Stage 1G netplan fix (match by driver virtio_net) use DHCP and applies it offline to a disposable copy. ADR 009 moves the qcow2 (about 2.7 GiB) to the node and uploads it through CDI into a 40 GiB Block-mode RWO gp3 PVC, where it is stored raw. qemu-guest-agent becomes an optional, offline pre-install. The source VM, golden artifact and Stage 1G images are unchanged. Details: Stage 1H record.
With a separate explicit approval for each gate, the Stage 1H design was executed on 2026-10-01. I1: a new standalone qcow2 was prepared offline from the kept virt-v2v output on conversion-host-01. Its netplan was replaced with a virtio_net match and DHCP, qemu-guest-agent was installed offline, and the virt-v2v first-boot scripts were removed (41 of 41 checks). The image was hashed (2b30e195...93b2) and made immutable. I2: that exact image booted under QEMU/KVM with UEFI through a disposable overlay and served the source page (54 of 54 checks, on the second run). I3: Terraform built 10 AWS resources in ap-south-1, including one m8i.xlarge with nested virtualization running CentOS Stream 9. On it, a single-node kubeadm cluster ran Kubernetes v1.36.5, CRI-O 1.36.6, flannel v0.28.9, KubeVirt v1.9.0, CDI v1.66.1 and the EBS CSI driver with an encrypted gp3 StorageClass. I4: the qcow2 was sent to the node over SFTP, with the same sha256 recorded on both sides. CDI imported it into a 40 GiB encrypted gp3 Block DataVolume, and it booted as the KubeVirt VM legacy-source-vm-migrated. The guest got 10.0.2.2 by DHCP through masquerade, kept its workload identity, connected its guest agent, and returned HTTP 200 with the 267-byte source page to the operator workstation through NodePort 30080. I5: a final end-to-end check passed. The VM, its disk (through the DataVolume, PVC, PV and CSI lifecycle) and the 10 Terraform-managed resources were then removed and verified gone in AWS, with no orphaned volume. The source VM was not contacted during Stage 1I and stayed in service. The migrated disk was the cold, point-in-time copy taken in Stage 1E, and no cutover was performed. Details: Stage 1I record, runtime snapshot, visual guide.
The Stage 1I teardown removed every AWS and Kubernetes resource the stage created; nothing from Stage 1I is running or billed in AWS. Stage 1I did not re-measure the ESXi host, so the host, management, datastore, nested-virtualization, time, access and maintenance-mode rows are the values last re-verified in Stage 1H. The guest VM, artifact and AWS rows reflect Stage 1I. Earlier versions of this section described the state after Stage 1H, when the target was a design that did not yet exist.
| Item | Value |
|---|---|
| Host | esxi-8-lab.localdomain, ESXi 8.0.3 build 24677879 |
| Management | vmk0 192.168.50.11/24 static, gateway and DNS 192.168.50.2 (unchanged) |
| Datastore | migration-datastore, VMFS-6, 199.8 GB, 155.5 GB free (166,960,562,176 bytes; GB = 2^30 bytes here, as in Stage 1A); holds legacy-source-vm/, kvm-learning-01/, conversion-host-01/ and iso/ (Ubuntu 24.04.4 and 24.04.5 ISOs) |
| Nested virtualization | HV Support 3; 64-bit L2 guest validated; KVM inside L2 validated (Stage 1B and again in conversion-host-01) |
| Time | NTP enabled and synchronized |
| Access | SSH key login via the esxi-8-lab alias; HTTPS 443 reachable |
| Guest VMs | 3, all powered on at the Stage 1I Gate I1 precheck. legacy-source-vm (Vmid 2), 192.168.50.31, nginx on :80, answered HTTP 200 with the source page after the Gate I5 teardown. conversion-host-01 (Vmid 3), 192.168.50.32. kvm-learning-01 (Vmid 1), 192.168.50.30, used in Stage 1I as the Terraform and kubectl jump host. No snapshots on any (as last checked in Stage 1H). Live power states can differ after operations outside the stages. |
| Conversion host | conversion-host-01 is the dedicated lab host for inspection, conversion and QEMU/KVM boot validation (ADR 006); ovmf 2024.02 added in Stage 1G; no QEMU guest running |
| Acquired source disk | Golden cold copy on the Windows host at C:\VMs\legacy-source-vm\stage-1e\ (outside Git, Windows read-only attribute since Stage 1G): descriptor + 40 GiB flat extent, sha256 72ca45c7...c9e7. Verified working copy on conversion-host-01 in /srv/migration-lab/working/ (read-only, immutable). |
| Converted artifacts | On conversion-host-01 in /srv/migration-lab/stage-1g/ (outside Git, read-only, hashed): qemu-img qcow2 (08c62ac5...5447), virt-v2v qcow2 + XML (94bc1cd6...91c6), network-remediated virt-v2v copy (5326b130...d3d6). None of these artifacts was edited. Stage 1I added the prepared image /srv/migration-lab/stage-1i/working/legacy-source-vm-prepared.qcow2 (sha256 2b30e195...93b2, 0444, immutable), which is the authoritative copy of what was migrated, plus the Stage 1I text evidence under /srv/migration-lab/stage-1i/evidence/. |
| AWS and Kubernetes | None. The Stage 1I target (10 Terraform-managed resources, the kubeadm cluster, KubeVirt, CDI, the migrated VM and its EBS volume) was destroyed in Gate I5 and verified absent in AWS. kvm-learning-01 keeps the Terraform configuration with an empty state, the retained evidence and virtctl; the temporary Stage 1I credentials and tunnels were removed |
| Maintenance mode | Disabled |
No stage after 1I is defined. Any further work, such as the controller-driven workflow of ADR 005, more workloads or an EKS comparison, would need its own definition and approval.
Stage 1I: section 28 of the Stage 1H record. Earlier entry criteria: Stage 1H in section 24 of the Stage 1G record, Stage 1G in section 21 of the Stage 1F record, Stage 1F in section 13 of the Stage 1E record, Stage 1E in section 14 of the Stage 1D record, Stage 1D in section 16 of the Stage 1C record, Stage 1C in section 11 of the Stage 1B record, Stage 1B in section 11 of the Stage 1A document.
