Stage 0 — Architecture Before Implementation
Understand the VMware → KubeVirt migration architecture before building it.
Visual first. Technical documentation later.
Status words keep their exact meaning: OBSERVED, INFERRED, DOCUMENTED, NOT TESTED, VALIDATED, UNVALIDATED, DECIDED, DEFERRED, UNKNOWN, BLOCKED. Colors always come with a label and a border style.
Five levels
Go as deep as you like. Level 1 alone gives you the big picture.
Stage 0 in 5 minutes
Twelve ideas. Click a card to jump to its diagram.
Jump to a concept
VMware engineer mode ROUGH ANALOGY
If you know vSphere, start here. These are rough analogies to help orientation, not equivalences. Diagrams with a VMware comparison now have a dashed amber border.
| VMware | Rough KubeVirt / KVM equivalent | See | |
|---|---|---|---|
| ESXi | ≈ | a Kubernetes node running Linux + KVM, managed by KubeVirt | diagram 04 |
| VMkernel | ≈ | the Linux kernel with the KVM module (/dev/kvm) | diagram 05 |
| VMX process | ≈ | the QEMU process inside the virt-launcher Pod | diagram 13 |
| vSwitch | ≈ | the node CNI dataplane + in-pod plumbing | diagram 19 |
| Port Group | ≈ | a networks[] entry: the pod network or a Multus NAD | diagram 19 |
| vNIC | ≈ | an interfaces[] entry: a virtio NIC with a binding | diagram 19 |
| PVSCSI | ≈ | virtio-blk or virtio-scsi | diagram 06 |
| VMXNET3 | ≈ | virtio-net | diagram 06 |
| VMDK | ≈ | a PVC, filled by CDI and stored raw | diagram 15 |
Understand the architecture
Stop here and you already know what Stage 0 is about.Big picture: one VM, two possible futures
Start with one Linux VM running nginx on ESXi. The question is how to move it to Kubernetes. The migration path moves the whole VM onto KubeVirt, where it is still a VM, now managed by Kubernetes. The modernization path repackages only the application as containers, so the VM disappears.
Why this matters: Most later decisions depend on which of the two paths you are on.
Remember this: MIGRATION = the VM remains a VM. MODERNIZATION = the VM dependency is removed.
Technical details
Stage 0 overview and the first demonstration, which is designed but not built. Read: architecture.md, migration-architecture.md, modernization-model.md.
VMware to KubeVirt: same guest, different stack
ROUGH ANALOGY VMware lens: ESXi + hostd become Kubernetes + KubeVirt. The VMX process becomes QEMU inside a Pod.
On the left, ESXi runs a VM with VMware virtual hardware. On the right, Kubernetes and KubeVirt manage a VirtualMachine object, and a virt-launcher Pod runs QEMU on KVM. Both stacks end with the same Ubuntu and nginx.
Why this matters: It shows what a migration must replace and what it must leave alone.
Remember this: Same guest OS. Different management and virtualization stack.
Technical details
How the two stacks are layered, and what KubeVirt adds to Kubernetes. Read: virtualization-fundamentals.md, kubevirt-architecture.md.
Three compute worlds
World A is a traditional VM on ESXi. World B is the same VM run by KubeVirt. World C is the application as a container, with no guest OS. Migration moves A to B; modernization moves A or B to C.
Why this matters: Without this split, "moving to Kubernetes" can mean two different projects at once.
Remember this: Migration = A → B. Modernization = A/B → C.
Technical details
The comparison table and the worked nginx example. Read: modernization-model.md.
Understand the technologies
KVM, QEMU, VirtIO and KubeVirt: what each one does.The virtualization stack
ROUGH ANALOGY VMware lens: ESXi VMkernel ≈ Linux + KVM. VMX process ≈ QEMU. hostd / vCenter ≈ Kubernetes API + KubeVirt.
Every VM platform has the same layers: hardware, hypervisor, virtual hardware, guest OS, application and a manager. VMware, plain KVM and KubeVirt-managed KVM differ only in the hypervisor, virtual hardware and manager rows.
Why this matters: It tells you which layer each migration problem lives in.
Remember this: Migration changes the lower machinery and the manager, not the guest.
Technical details
Hypervisor types, and how ESXi and KVM compare layer by layer. Read: virtualization-fundamentals.md.
KVM, QEMU and libvirt: three jobs, one VM
ROUGH ANALOGY VMware lens: VMkernel's VMM ≈ KVM. VMX process ≈ QEMU. hostd ≈ libvirt. ESXi does all three as one product.
KVM is a Linux kernel module that runs guest code on the real CPU using VT-x or AMD-V. QEMU is one user-space process per VM that builds the machine: firmware, chipset and devices. libvirt describes the VM as domain XML and starts QEMU.
Why this matters: Every KubeVirt VM is this exact stack inside a Pod. Debugging starts with knowing which layer failed.
Remember this: KVM runs the VM; QEMU provides the machine; libvirt manages it.
Technical details
KVM, QEMU and libvirt in detail, and how KubeVirt uses them. Read: virtualization-fundamentals.md, kubevirt-architecture.md.
VirtIO: the hardware changes under the guest
ROUGH ANALOGY VMware lens: PVSCSI → virtio-blk / virtio-scsi. VMXNET3 → virtio-net.
VMware guests use PVSCSI disks and VMXNET3 NICs. Under KVM the guest sees virtio-blk/scsi and virtio-net. The drivers must be in the initramfs for the guest to boot. qemu-img changes only the disk format; virt-v2v also adapts the guest.
Why this matters: New hardware under an unchanged guest is the usual reason a converted VM fails to boot or has no network.
Remember this: The disk bytes move; the hardware under them changes.
Technical details
Device models, and where guest conversion fits in the flow. Read: virtualization-fundamentals.md, migration-architecture.md.
What is KubeVirt?
KubeVirt is a Kubernetes add-on. It adds new object types such as VirtualMachine, controllers that act on them, and a node agent. A VM becomes something you kubectl apply, protect with RBAC and schedule like a Pod. It does not replace KVM; it drives QEMU and KVM from inside Pods.
Why this matters: Everything else in Stage 0 builds on this one idea.
Remember this: KubeVirt makes VMs manageable through Kubernetes.
Technical details
KubeVirt components, objects and what the nodes need. Read: kubevirt-architecture.md.
KubeVirt component map
virt-operator installs and upgrades KubeVirt. virt-api provides admission webhooks and subresources such as start, stop and console. virt-controller turns VMs into VMIs and virt-launcher Pods. virt-handler runs on every VM node and tells virt-launcher to define and start the libvirt domain.
Why this matters: When a VM will not start, the failing component tells you where to look.
Remember this: Cluster brain: virt-controller. Node hands: virt-handler.
Technical details
The component table and the full lifecycle of starting a VM. Read: kubevirt-architecture.md.
VM vs VMI
The VirtualMachine is the persistent definition: it exists while the VM is stopped and holds runStrategy. The VirtualMachineInstance is one running instance. virt-controller creates it when the VM should run and deletes it when it stops.
Why this matters: Lifecycle questions, such as what survives a stop or what restarts a crash, depend on this split.
Remember this: VM = specification. VMI = running instance.
Technical details
VM, VMI and runStrategy. Read: kubevirt-architecture.md.
Understand Kubernetes control
CRDs, controllers, reconciliation and status.CRD → controller → VM
A CRD teaches the API server a new object type, such as VirtualMachine. Creating an object of that type only stores data. A controller watches those objects, compares desired with actual state, and creates or updates resources until they match. Status flows back to the object.
Why this matters: The planned migration platform is exactly this pattern: a new CRD plus our own controller.
Remember this: CRD = defines a new API type. Controller = implements the behavior.
Technical details
CRDs, controllers, informers and work queues. Read: crd-controller-fundamentals.md.
The reconciliation loop
A controller does not run a script once. It observes, compares, takes one idempotent step, then observes again. Example: with runStrategy: Always, if the VM crashes, virt-controller creates a new VMI. That is documented KubeVirt behavior; Stage 0 did not test it.
Why this matters: Idempotent, restart-safe steps are what make a long migration survivable.
Remember this: Controller = state convergence.
Technical details
The reconcile loop, idempotency and requeueing. Read: crd-controller-fundamentals.md.
Spec, status and conditions
spec is what you want; only you write it. status is what actually happened; only the controller writes it. The designed migration CR reports independent conditions such as SourceDiscovered, DiskImported and Ready, each with a reason, a message and observedGeneration.
Why this matters: You debug a controller by reading status, so status must be honest and specific.
Remember this: The controller reports what it knows.
Technical details
status.phase, conditions and observedGeneration, as designed. Read: crd-controller-fundamentals.md, migration-architecture.md.
The KubeVirt resource chain: who owns this?
ROUGH ANALOGY VMware lens: VM in inventory ≈ VirtualMachine. VMX process ≈ QEMU in virt-launcher. VMkernel's VMM ≈ KVM.
Seven layers, each with a different owner. You (or the migration controller) own the VirtualMachine. virt-controller owns the VMI and creates the virt-launcher Pod. virt-handler has the libvirt domain defined. libvirt runs QEMU, the node kernel provides KVM, and the application team owns the guest.
Why this matters: "Who owns this layer?" is the fastest way to find the right logs.
Remember this: VM → VMI → virt-launcher Pod → libvirt → QEMU → KVM → guest.
Technical details
The start-up sequence, step by step. Read: kubevirt-architecture.md.
Understand migration mechanics
Storage, CDI, the source model, the flow, the state machine and networking.Storage object stack
ROUGH ANALOGY VMware lens: VMDK ≈ PVC. Datastore + storage policy ≈ StorageClass. Datastore upload ≈ DataVolume import.
The guest sees a virtio disk. It comes from a PVC, which a DataVolume filled. A StorageClass is a policy that tells a CSI driver to create one real volume per claim, for example an EBS gp3 volume on AWS.
Why this matters: Storage settings such as access mode and zone decide what the VM can and cannot do later.
Remember this: One disk = one PVC. A StorageClass is a policy, not a pool you browse.
Technical details
PVCs, StorageClasses, CSI and the planned AWS storage. Read: cdi-storage-model.md.
CDI data path
CDI takes a disk image (qcow2, VMDK or raw), imports or receives it, converts it to raw and writes it into a PVC, reporting progress on a DataVolume. CDI ends when the DataVolume is Succeeded. KubeVirt begins when a VM references the PVC.
Why this matters: A clear boundary tells you which system to debug when a migrated VM has a disk problem.
Remember this: CDI imports / prepares disk into Kubernetes storage. CDI does not run VMs.
Technical details
CDI sources, DataVolume phases and scratch space. Read: cdi-storage-model.md.
The VMware source model
ROUGH ANALOGY VMware lens: These are the objects you know from the Host Client: inventory, .vmx, .nvram, VMDK, vNIC, port group.
A VMware VM is three kinds of information. Metadata (inventory, .vmx, firmware, NICs) is small and is translated into a VM spec. Disk data (the VMDK) is large and is copied and converted after power-off. Runtime state (power, IP lease, VMware Tools) is only observed.
Why this matters: Each kind is handled at a different time, by a different step.
Remember this: Metadata is translated, disk data is copied, runtime state is only observed.
Technical details
The source model and what a free ESXi host exposes. Read: vmware-source-model.md.
Migration architecture
The designed cold-migration road has three rows. On the source side the VM is discovered, assessed and powered off, and its disk is copied. The disk is then converted and staged. On the Kubernetes target, CDI imports it and a KubeVirt VM is created and validated. A future controller sits on top.
Why this matters: This is the whole project on one page. Each box later becomes a controller phase.
Remember this: Stage 0 designed the whole road; nothing on it was built in Stage 0.
Technical details
The flow table, cold vs warm, MTV as reference and the CRD sketch. Read: migration-architecture.md, ADR 003, ADR 005.
Migration state machine
The designed status.phase goes from Pending through Discovering, Validated, Preparing, Converting, Importing, CreatingVM, Starting and Validating to Completed. Transient errors go to Retryable and resume the same phase. A human decision goes to ManualInterventionRequired. Permanent errors go to Failed.
Why this matters: Checkpoints make a half-finished migration resumable instead of a mystery.
Remember this: State machine = reconciliation checkpoints.
Technical details
Per-phase desired and observed state, transition types and idempotency rules. Read: migration-state-machine.md.
Networking model
ROUGH ANALOGY VMware lens: vNIC ≈ virtio NIC in interfaces[]. Port group ≈ networks[] entry. vSwitch ≈ CNI dataplane + in-pod plumbing.
In KubeVirt, networks[] says where a NIC connects (the pod network or a Multus network), and interfaces[] says how (the binding). The Stage 0 plan is one NIC on the pod network with masquerade, reached through a Service. Bridge and Multus are optional and deferred.
Why this matters: Networking changes more in a migration than anything else, including the VM's IP address.
Remember this: networks[] = where. interfaces[] = how.
Technical details
Bindings, Multus, NADs and the ESXi comparison. Read: networking-model.md.
Network mapping
ROUGH ANALOGY VMware lens: Port group + VLAN + vNIC settings are the source of the mapping.
Every NIC needs five answers: destination network, binding, IP addressing, MAC address and guest OS configuration. With masquerade, the guest must use DHCP, so a static VMware IP will not work as-is. The interface name can change (ens192 to enp1s0), which can break netplan.
Why this matters: A VM that boots but has no network is still a failed migration.
Remember this: Network migration is a mapping problem, not simply a NIC swap.
Technical details
The mapping rules and the demo plan. Read: networking-model.md.
Understand engineering decisions
What is decided, what is proven, and what comes next.Migration vs modernization
Migration moves the whole machine and ends as a KubeVirt VirtualMachine; the guest OS is kept and still patched like a VM. Modernization moves only the application into a container image run by a Deployment, and the guest OS goes away.
Why this matters: Treating a migrated VM as modernized hides the guest OS you still have to operate.
Remember this: Migration keeps the VM; modernization removes it.
Technical details
The comparison table and the nginx worked example. Read: modernization-model.md.
Feasibility decision map
Stage 0 sorted every finding into five categories. Validated means we saw it work. Documented means official docs say so, but we did not run it. Unvalidated means unknown. Blocked means impossible in the current lab mode. Deferred means intentionally not decided yet.
Why this matters: It stops paper decisions from being mistaken for proven facts.
Remember this: Architecture decisions are different from proven runtime facts.
Technical details
The full feasibility table, compatibility tuple and host requirements. Read: feasibility.md, evidence-index.md.
Stage 0 decisions: ADR 001 to 005
Five accepted decisions answer the big questions. 001: the source is a local nested ESXi. 002: the target is AWS, Terraform-only and disposable. 003: cold before warm. 004: upstream KubeVirt + CDI. 005: a custom controller owns the workflow, with scripts only as building blocks.
Why this matters: ADRs record why, so later stages do not quietly re-decide.
Remember this: Where from, where to, how, onto what, who orchestrates.
From Stage 0 to Stage 1
Stage 0 produced concepts, designs, a feasibility table and five ADRs, and built nothing. Stage 1 then built selected parts in the lab: a stable ESXi host, a source VM, a cold disk copy, a conversion host, and a converted guest booting on QEMU/KVM. 1H designed the KubeVirt target. 1I built it temporarily on AWS (a single-node kubeadm cluster on EC2), migrated the VM, validated it and destroyed the target on 2026-10-01. The diagram was drawn before 1I.
Why this matters: It shows where design ends and evidence begins.
Remember this: Stage 0 asks: What should we build? Stage 1 asks: Can we actually make the important parts work?
Technical details
Stage 0 exit criteria and the Stage 1 record. Read: Stage 0 README, Stage 1 README, Stage 1 Visual Learning.
What was built, and what is not?
Design, proof and future work are three different things.STAGE 0 DESIGN
on paper only
- VirtualMachineMigration CRD + controller
- Migration state machine and conditions
- Network mapping rules (pod network + masquerade)
- CDI import flow and storage mapping
- Disposable AWS target (ADR 002)
STAGE 1 PROVEN
observed in the lab
- ESXi survives a controlled reboot (1A)
- Nested KVM learning VM (1B)
- VMware source VM with nginx (1C)
- Cold disk acquisition, sha256 verified (1E)
- qemu-img + virt-v2v conversion, QEMU/KVM boot with UEFI + virtio (1G)
- Guest network fix on a disposable copy (1G)
- Temporary single-node AWS cluster with KubeVirt + CDI, built with Terraform and kubeadm, then destroyed (1I)
- The migrated VM running on KubeVirt and serving the source page (1I)
FUTURE
not built
- The migration controller and CRD (ADR 005)
- Application modernization (container path)
- Warm or live migration, multi-node clusters, EKS
Stage 0 interview visuals
From interview-notes.md. Click a question.Q1 How would you migrate a VMware VM to Kubernetes?
Rehost it as a KubeVirt VM: discover and assess, power off (cold), convert the disk and guest drivers, import the disk into a PVC with CDI, create a VirtualMachine with storage and network mappings, boot it and validate the app.
Q2 What is KubeVirt?
A Kubernetes add-on that runs and manages VMs as Kubernetes objects, next to containers, using the Kubernetes API, scheduler, networking and storage.
Q3 What is the difference between a VM and a VMI?
The VirtualMachine is the persistent definition and desired run state; the VMI is one running instance of it. Like a StatefulSet with one replica versus its Pod.
Q4 How does KubeVirt run a VM?
VirtualMachine, then virt-controller creates a VMI and a virt-launcher Pod, the scheduler places it, virt-handler has libvirt start the domain, and QEMU runs the VM on /dev/kvm.
Q5 Where does QEMU run?
In user space, inside the compute container of the VM's virt-launcher Pod, on the node where that Pod was scheduled. One QEMU process per VM.
Q6 What is KVM?
A Linux kernel module that turns Linux into a hypervisor using VT-x / AMD-V. It exposes /dev/kvm, which QEMU uses to create VMs and run vCPUs.
Q7 What does CDI do?
It fills PVCs with VM disk data: import, upload, clone or blank. It converts images to raw and reports progress on a DataVolume. It knows nothing about CPUs or NICs.
Q8 How would you design a migration CRD?
One namespaced CR per VM migration: a declarative spec (source, destination, strategy, storage and network mapping, validation) and a controller-owned status (phase, conditions, progress, inventory, errors).
Q9 What happens when migration fails halfway?
The CR keeps the phase and records the error. Transient errors retry in the same phase; permanent ones go to Failed or ManualInterventionRequired. The source disk is intact, so the source can be powered back on.
Q10 What is the difference between VM migration and application modernization?
Migration moves the whole VM, guest OS included, without changing the app. Modernization repackages the app, for example as a container in a Deployment, which removes the VM and guest OS.
The whole map
Stage 0 gives us the map.
Stage 1 proves selected roads on the map.
5 things to remember
KubeVirt manages VMs through Kubernetes.
KVM runs the VM; QEMU provides the machine.
CRDs describe desired state; controllers reconcile it.
CDI puts disk content into Kubernetes storage.
Migration keeps the VM; modernization removes it.
- Stage 0 README
- architecture.md
- virtualization-fundamentals.md
- kubevirt-architecture.md
- crd-controller-fundamentals.md
- cdi-storage-model.md
- vmware-source-model.md
- migration-architecture.md
- migration-state-machine.md
- networking-model.md
- modernization-model.md
- feasibility.md
- evidence-index.md
- interview-notes.md
- glossary.md
- ADRs 001 to 005