Request section 24. Answers are written to be said out loud in a Kubernetes / platform engineering interview: a short answer first, then the detail an interviewer will probe. Deeper explanations are linked.
Short answer: Rehost it as a KubeVirt VM: discover and assess the VM, power it off (cold) or precopy with CBT (warm), convert the disk and guest drivers for KVM, import the disk into a PVC with CDI, create a KubeVirt VirtualMachine that uses that PVC with network and storage mappings, boot it, and validate the application. At scale, use a tool like MTV/Forklift; the underlying steps are the same.
Detail:
Short answer: A Kubernetes add-on that lets you run and manage virtual machines as Kubernetes objects, next to containers, using the Kubernetes API, scheduler, networking and storage.
Detail: It adds CRDs (VirtualMachine, VirtualMachineInstance, and others), cluster controllers (virt-controller, virt-api, virt-operator) and a node daemon (virt-handler). Each running VM is a QEMU process managed by libvirt inside a normal Pod (virt-launcher), accelerated by KVM on the node. It is a CNCF project; OpenShift Virtualization is Red Hat’s distribution of it. See kubevirt-architecture.md.
Short answer: The VirtualMachine is the persistent definition and desired run state; the VirtualMachineInstance is one running instance of it. Like a StatefulSet with one replica versus its Pod.
Detail: The VM survives stop/start and holds spec.template and spec.runStrategy (Always, RerunOnFailure, Manual, Halted). virt-controller creates a VMI from the template when the VM should run and deletes it when the VM stops. The VMI carries live status: phase, node, IPs, guest info. You can create a bare VMI, but nothing restarts it.
Short answer: VirtualMachine -> virt-controller creates a VMI -> virt-controller creates a virt-launcher Pod -> the scheduler places it -> virt-handler on that node tells libvirt inside the Pod to start the domain -> QEMU runs the VM using /dev/kvm.
Detail: virt-api validates the object at admission. The launcher Pod requests CPU, memory (plus overhead) and the devices.kubevirt.io/kvm resource, mounts the disk PVCs, and gets networking from CNI/Multus. virt-handler renders domain XML from the VMI and keeps the domain and VMI status in sync. Status flows back: domain -> VMI phase -> VM printableStatus.
Short answer: In user space, inside the compute container of the VM’s virt-launcher Pod, on the node where that Pod was scheduled. One QEMU process per VM.
Detail: Because it is inside a Pod, it is subject to the Pod’s cgroups (CPU/memory limits), network namespace and mounted volumes. The guest’s vCPUs are QEMU threads. The kernel part of virtualization is KVM on the node; QEMU talks to it through /dev/kvm. If the Pod dies, the VM dies, which is why the VM object and runStrategy exist.
Short answer: A Linux kernel module that turns Linux into a hypervisor by using hardware virtualization (Intel VT-x / AMD-V, plus EPT/NPT for memory). It exposes /dev/kvm, which QEMU uses to create VMs and run vCPUs.
Detail: KVM does CPU and memory virtualization; QEMU does device emulation; libvirt does management. Unlike ESXi, where VMkernel is a purpose-built hypervisor OS, KVM is part of a general-purpose Linux kernel, so the same node can run containers and VMs. In the cloud you need bare metal or nested virtualization to get /dev/kvm.
Short answer: With CDI. Either put the VMDK (or a qcow2 converted from it) at an HTTP/S3 URL and create a DataVolume with an http or s3 source, or push it with virtctl image-upload. CDI converts it to raw in a new PVC, and the VM references that PVC.
Detail:
monolithicSparse disk is one file. Copying only the descriptor gives you nothing.qemu-img check.vddk source (needs the VDDK image and API access; not viable on free ESXi).Short answer: It populates PVCs with VM disk data: import from URLs, registries, S3 or VMware (VDDK); upload from a client; clone from another PVC or snapshot; or create blank disks. It converts images to raw and reports progress through the DataVolume CR.
Detail: CDI ends when the DataVolume is Succeeded and the PVC holds a bootable raw disk. KubeVirt begins when a VM references that PVC. CDI knows nothing about CPUs or NICs. See cdi-storage-model.md.
Short answer: One namespaced CR per VM migration, with a declarative spec (source, destination, strategy, storage mapping, network mapping, validation) and a controller-owned status (phase, conditions, progress, frozen source inventory, created destination objects, errors, timestamps). Credentials by Secret reference only.
Detail:
status.phase drives the state machine; conditions give independent, typed facts.observedGeneration shows whether status reflects the latest spec.powerOffPolicy, because powering off the source is the only action that affects production.Short answer: It watches VirtualMachineMigration objects and their children, and for each one runs a reconcile: read the current phase, perform the next idempotent step for that phase (power off, convert, create DataVolume, create VM), observe the result, update status, and requeue until a terminal phase.
Detail: Events go to a work queue keyed by namespace/name, which deduplicates and ensures one worker per object. Long steps are not awaited in-process; the controller creates Jobs or DataVolumes and requeues or reacts to watch events when their status changes. Status updates go through the status subresource. See crd-controller-fundamentals.md.
Short answer: The CR keeps the phase where it failed and records the error. Transient errors are retried with backoff in the same phase; permanent or ambiguous errors go to Failed or ManualInterventionRequired. Because the source disk is only read after power-off, the source is intact and can be powered back on.
Detail:
Short answer: Make every step “ensure”, not “do”: deterministic names for created objects, create-if-absent and update-if-different, check external state before acting, record evidence (checksums, URLs, object names) in status before advancing the phase, and treat “already done” and “not found on delete” as success.
Detail:
<migration>-disk-<n>; staging key <migration-uid>/disk-<n>.qcow2.resourceVersion) on status updates and requeue on conflict.Short answer: Migration (rehost) moves the whole VM, guest OS included, to a new platform without changing the app. Modernization changes how the app is packaged and run, for example as a container image in a Deployment, which removes the VM and guest OS.
Detail: KubeVirt migration is fast and low-risk and puts VMs on the same platform as containers, but you still patch and operate a guest OS. Modernization needs assessment and app changes but gives horizontal scaling, rolling updates and image-based delivery. A good program migrates first and modernizes selectively. Project 1.5 shows both on the same nginx workload. See modernization-model.md.
Answers below are backed by what was actually run in Stage 1B on kvm-learning-01 (see Stage 1B record, nested KVM feasibility). Question 6 above gives the short definition of KVM; question 14 adds the lab view.
Short answer: The hypervisor built into the Linux kernel: the modules kvm and kvm_intel (or kvm_amd). It uses VT-x/AMD-V and EPT/NPT to run guest code directly on the CPU, and exposes /dev/kvm so a user-space program can create VMs and vCPUs.
Detail: KVM handles CPU and memory virtualization and a few latency-critical devices (local APIC, timers), and it handles VM exits. It emulates no disks, NICs or firmware. In the lab, Ubuntu autoloaded kvm_intel inside an ESXi guest as soon as the (virtual) CPU showed the vmx flag, with nested=Y and ept=Y; no tuning was needed.
Short answer: A user-space machine emulator and virtual machine monitor. One qemu-system-x86_64 process per VM provides the chipset, firmware, PCI bus and every virtual device, and runs guest code either through KVM or with its own software emulator (TCG).
Detail: Each vCPU is a thread of the QEMU process (CPU 0/KVM, or CPU 0/TCG without KVM). Guest RAM is ordinary memory in the QEMU process. Disks and NICs are split into a backend (qcow2 file, user-mode network) and a frontend device model (virtio-blk, virtio-net). The machine model (q35, pc) picks the chipset. In KubeVirt, the same QEMU runs inside the virt-launcher Pod.
Short answer: They do different jobs. KVM makes the CPU run guest code safely at native speed but cannot build a whole machine; QEMU builds the whole machine but, alone, has to emulate the CPU in software. Together you get a full VM at near-native CPU speed.
Detail: In the lab, the same tiny guest ran a 128 MiB md5sum in 0.22 s with QEMU + KVM, versus 2.05 s with QEMU alone (TCG), and a shell loop in 0.33 s versus 5.09 s. Device emulation was QEMU’s job in both runs. KVM cannot do anything useful without a user-space VMM; QEMU without KVM works, only slowly. They are separate components from separate projects.
Short answer: The character device (major 10, minor 232) through which user space talks to KVM. QEMU opens it and uses ioctl() calls to create a VM, create vCPUs, register memory and run vCPUs (KVM_RUN).
Detail: It exists only when a KVM module is loaded, which requires VT-x/AMD-V visible to the kernel. Permissions come from the kvm group (crw-rw---- root kvm). In the lab it appeared inside the ESXi guest before any package was installed. The running QEMU process held /dev/kvm plus anon_inode:kvm-vm and anon_inode:kvm-vcpu:0 file descriptors. In KubeVirt, virt-handler advertises it to the scheduler as the devices.kubevirt.io/kvm resource.
Short answer: QEMU sets up the VM through /dev/kvm, then each vCPU thread calls ioctl(KVM_RUN). KVM enters the guest with VMLAUNCH/VMRESUME and the guest runs natively until a VM exit. KVM handles the exit in the kernel if it can; otherwise it returns to QEMU, which emulates the device access and calls KVM_RUN again.
Detail: Typical exits are port or MMIO I/O, some MSR accesses, HLT, interrupts and EPT faults. The cost of a VM exit is what separates good from bad virtual performance. In the lab, three hypervisors deep, one exit travels through Workstation, ESXi and KVM. CPU-bound work in the L3 guest was as fast as in L2, but the exit-heavy firmware and boot phase took about 17 s under KVM, compared with about 6 s under TCG.
Short answer: Running a hypervisor inside a virtual machine. The outer hypervisor must expose (emulate) the CPU’s virtualization extensions, VT-x and EPT, to the inner one, so that the inner hypervisor can run its own hardware-accelerated guests.
Detail: VMware calls the setting VHV (vhv.enable = "TRUE"). KVM calls it the nested parameter of kvm_intel. AWS enables it per instance on supported instance types. It works, but every exit of the innermost guest is handled by the outermost hypervisor first and reflected inward, so it costs performance. The lab stacks three hypervisors: Workstation, ESXi, KVM.
Short answer: L0 is the hypervisor on the physical hardware. L1 is a guest hypervisor running as a VM on L0. L2 is a VM run by L1. That is the Linux kernel’s naming for nested guests.
Detail: In our lab: L0 is the VMware Workstation VMM on the laptop, L1 is ESXi, L2 is kvm-learning-01, and because L2 runs KVM there is also an L3 test VM. Windows is the host OS of a type-2 hypervisor and has no level number. On AWS with nested virtualization, Nitro is L0, the EC2 worker node running KVM is L1, and the KubeVirt VM is L2. On a .metal instance, the node’s own KVM becomes L0.
Short answer: A management layer for hypervisors: an API, a daemon (libvirtd, or modular daemons such as virtqemud) and a declarative VM format (domain XML). For QEMU/KVM it generates the QEMU command line, starts and monitors the QEMU process, and applies security labels, cgroups and logging.
Detail: In the lab, a hand-written 30-line domain XML became a QEMU command line with -accel kvm -cpu host,migratable=on, a QMP monitor socket, -sandbox, -blockdev nodes, five PCIe root ports and a USB controller the XML never mentioned. QEMU ran as user libvirt-qemu under an enforcing AppArmor profile. KubeVirt runs one libvirtd per VM inside the virt-launcher Pod and generates the domain XML from the VMI.
Short answer: The command-line client for the libvirt API. It sends requests to the libvirt daemon (define, start, shutdown, destroy, list, dumpxml); it does not run VMs itself.
Detail: The connection URI matters. qemu:///system talks to the system daemon (root-owned VMs); qemu:///session is a per-user instance. In the lab, virsh uri returned qemu:///session for the normal user and qemu:///system for root. The VMware counterpart is roughly vim-cmd talking to hostd.
Short answer: An OASIS-standard family of paravirtual devices for VMs. The guest knows it is virtual and exchanges requests with the hypervisor through shared-memory queues (virtqueues) rather than through emulated hardware registers. Main types: virtio-net, virtio-blk, virtio-scsi.
Detail: VirtIO devices appear as PCI devices with vendor 1af4: transitional IDs (1000 net, 1001 blk, 1004 scsi) or modern IDs (1041, 1042, 1048). The lab saw both, depending on where QEMU or libvirt placed the device. Linux ships the drivers in the kernel (virtio_blk, virtio_scsi, virtio_net). Windows needs the virtio-win drivers. KubeVirt presents VirtIO devices by default.
Short answer: Both are paravirtual NICs with the same job, but they are different devices. VMXNET3 is VMware’s (PCI 15ad:07b0, driver vmxnet3); virtio-net is the KVM/QEMU standard (PCI 1af4:1000 or 1041, driver virtio_net). A guest must have the right driver, and its network configuration must not be tied to the old device.
Detail: After conversion the NIC changes driver, MAC vendor prefix (00:0c:29 to 52:54:00) and usually its interface name (ens192 to enp1s0). In the lab VM, netplan matches the NIC by driver: "vmxnet3", so after a naive disk move the VM would boot with no configured network. virt-v2v or a pre-migration step must fix that.
Short answer: PVSCSI is VMware’s paravirtual SCSI controller (PCI 15ad:07c0, driver vmw_pvscsi). The VirtIO equivalents are virtio-scsi (also a SCSI controller, disks stay /dev/sdX) and virtio-blk (a simpler per-disk device, disks become /dev/vdX). Same role, different devices and drivers.
Detail: KubeVirt uses virtio-blk by default for disks, so a migrated Linux guest usually goes from /dev/sda to /dev/vda. The lab VM mounts / and /boot/efi by UUID and boots with root=UUID=..., so it survives the rename. A guest with /dev/sda1 in /etc/fstab would not. The Ubuntu kernel has the VirtIO storage drivers built in, so no initramfs rebuild is needed; older or custom kernels may need one.
Short answer: QEMU cannot use hardware acceleration. -accel kvm fails, and QEMU can only run with TCG software emulation, which is roughly 10x slower for CPU-bound work (9x to 15x in the lab). KubeVirt will not schedule VMs on that node unless software emulation is explicitly enabled, which is only for testing.
Detail: Diagnose from the bottom up:
vmx/svm in /proc/cpuinfo? If not, virtualization is disabled in firmware, or on a VM the outer hypervisor is not exposing it (VHV off, or a cloud instance without nested virtualization).lsmod | grep kvm)? Does dmesg show an error such as “disabled by bios”?kvm group?kvm-ok summarizes these checks. Never treat a TCG run as proof that KVM works: check for the CPU n/KVM thread, the /dev/kvm file descriptor, or Hypervisor detected: KVM in the guest.
Answers below are backed by the Stage 1C source VM legacy-source-vm (see Stage 1C record). Statements about the KubeVirt side are INFERRED; nothing has been migrated yet.
Short answer: Record three things before touching it: the platform view (hypervisor config), the guest view (OS, devices, identifiers, network, services), and a deterministic application check. Then the migrated copy can be compared fact by fact instead of “it seems to work”.
Detail: For legacy-source-vm the baseline covers:
.vmx path, CPU/RAM, thin 40 GB disk, PVSCSI, VMXNET3, MAC, firmware, snapshots.lsblk/blkid UUIDs, fstab, kernel command line, interface name and driver, netplan, routes, DNS, loaded drivers, EFI boot entries, cloud-init state.Collect it with commands whose output can be diffed. Label what was observed versus inferred.
Short answer: Everything stored on the disk survives: filesystem UUIDs, PARTUUIDs, the GPT disk GUID, machine-id, SSH host keys, hostname, config files. Everything the platform presents changes: controller and NIC drivers, device names, PCI paths, interface name, MAC, SMBIOS UUID and serial, and UEFI NVRAM entries.
Detail: In the lab, /dev/sda behind PVSCSI becomes /dev/vda on virtio-blk, ens192/vmxnet3 becomes something like enp1s0/virtio_net, and the VMware MAC 00:0c:29:0f:3d:15 is replaced unless the KubeVirt spec sets macAddress. Some platform identifiers can be pinned in KubeVirt if the guest depends on them: MAC, firmware.uuid, firmware.serial. Device names cannot and should never be treated as identity.
Short answer: Its network config is tied to the old NIC. Here, netplan configures the interface named ens192. On KubeVirt the virtio-net NIC gets a different name, nothing matches it, and the interface stays unconfigured.
Detail: The rendered networkd file contains [Match] Name=ens192, and the name comes from VMware’s PCI slot 192. The Stage 1B VM had the same problem with match: driver: vmxnet3. Fixes belong to the migration design, not the source: match by MAC and preserve the MAC, rewrite the config during conversion (virt-v2v does some of this), or supply target-side config. Even with a matching interface, a static 192.168.50.x address does not fit KubeVirt’s default masquerade pod network. Keeping the address needs a bridged secondary network.
Short answer: Because device names depend on the controller and discovery order, while filesystem UUIDs are stored inside the filesystem and travel with the disk.
Detail: legacy-source-vm mounts / and /boot/efi by /dev/disk/by-uuid/..., boots with root=UUID=db8b3bb3-..., and GRUB finds its files with search --fs-uuid. The switch from PVSCSI (sda) to virtio-blk (vda) therefore needs no edit. /dev/disk/by-path changes with the PCI topology, and /dev/disk/by-id is empty on this VMware disk, so neither is a good anchor. A guest with /dev/sda1 in fstab would drop to an emergency shell after migration.
Short answer: The boot entries are not on the disk. VMware keeps them in the VM’s .nvram file, so the target starts with an empty or different variable store and must find the loader through the removable-media fallback path \EFI\BOOT\BOOTX64.EFI.
Detail: On legacy-source-vm that file exists and is identical to shimx64.efi. On a fallback boot, shim runs fbx64.efi, which recreates the “Ubuntu” entry from BOOTX64.CSV. Two more things to settle in the target spec:
A BIOS-to-UEFI or UEFI-to-BIOS mismatch is a different, harder failure.
Short answer: Each has a different job and a different blast radius. The source VM must stay a clean, unmodified reference. The learning VM is where experiments and package installs happen. A conversion host needs disk access and conversion tooling. Mixing them contaminates the baseline or quietly makes an architecture decision.
Detail: In the lab, kvm-learning-01 (KVM/QEMU/libvirt installed, nested virtualization on) was powered off after the source baseline and never touched legacy-source-vm. The conversion-host decision stays explicitly deferred (local helper, AWS helper or another runtime) instead of being made implicitly by reusing the learning VM.
Answers below are backed by the read-only Stage 1D investigation of legacy-source-vm (see Stage 1D record).
Short answer: A .vmx (configuration), an .nvram (firmware variables), the virtual disk (a descriptor .vmdk plus its data extent), a few small metadata files (.vmxf, .vmsd), and runtime files (.vswp, locks, logs). A migration needs the disk contents. The .vmx is only a reference for writing the target VM definition, and the .nvram is not moved at all.
Detail: In the lab folder there are 14 files, 7.5G in total. About 4.1 GiB of that is swap that exists only while the VM runs, and 3.4 GiB is the allocated disk data. The target KubeVirt VM is a new object: its spec is written from the .vmx facts (CPU, RAM, EFI, disk bus, NIC, optionally MAC), its disk is imported from the extent, and its firmware state is created fresh. It then boots through the ESP fallback loader.
Short answer: Usually two files. A small text descriptor (createType="vmfs", CID, extent list, ddb.* bookkeeping) and a -flat.vmdk extent that is simply the raw disk sectors. Thin provisioning is done by VMFS (holes in the flat file), not by the VMDK format.
Detail: The lab disk’s descriptor is 541 bytes and names one extent of 83,886,080 sectors. The flat file has a logical size of 40 GiB but only 3,487 1 MiB blocks allocated (vmkfstools -D: nb 3487). parentCID=ffffffff shows there is no snapshot chain. The descriptor’s adapterType and virtualHWVersion are creation-time hints and do not describe the running VM. Other layouts exist: Workstation’s monolithicSparse, the 2gbsparse export format, OVF’s streamOptimized. Never assume one; read the descriptor.
Short answer: Without vCenter and with a restricted API, use the host’s SSH: power the VM off cleanly, then scp the descriptor and flat extent, or first vmkfstools -i ... -d 2gbsparse to a staging folder and copy the compact result. Verify with a checksum on both ends.
Detail: In the lab, scp streamed at about 137 MiB/s, so the raw 40 GiB copy takes about 5 minutes. A 2gbsparse export would move only about 3.5 GiB, but it re-encodes the container and needs a VMDK-aware tool to verify. The HTTPS /folder file service also works with an authenticated session, but brings no advantage over the existing SSH key. OVF export depends on the API and was not pursued. Tools like MTV/Forklift use VDDK or NFC and assume vCenter or a licensed API.
Short answer: Two reasons. VMFS holds an exclusive lock on an open disk, so you cannot even read it. And if you could, the copy would be taken from a live, changing filesystem: crash-consistent at best. Hot copies need a snapshot, which freezes the base disk and redirects writes to a delta.
Detail: In the lab, a 1 MiB dd read of the running flat file failed with “Device or resource busy”, and the HTTPS file service returned HTTP 500 for it. Snapshots are forbidden for this source VM, so the only consistent option is a cold copy after a graceful guest shutdown. Warm migration tools use snapshots plus changed block tracking (CBT); CBT is off on this VM.
Short answer: With the VM powered off, hash the flat extent on the host before and after the copy, and compare the copy’s hash with it. Cheap supporting checks: flat file size and mtime, VMFS allocation (nb) and file address, descriptor/.vmx/.nvram checksums, power-state history and snapshot count.
Detail: In the Stage 1E acquisition, a full 40 GiB hash took about 2.5 minutes per pass on ESXi. All three hashes matched: source before the copy, the copy, and source after the copy. The descriptor CID turned out not to be a write indicator on this ESXi disk: it stayed the same across a run that wrote to the disk. The cheap checks are strong indicators but not proof. The hash proves copy fidelity, not filesystem consistency; that comes from the clean shutdown. Once the source is powered on again, its disk changes legitimately (journal, logs), so “unchanged” can only be claimed for the powered-off window. After conversion, disk hashes no longer apply: verify the migrated VM at guest and application level (FS UUIDs, machine-id, SSH host keys, the nginx page hash).
Answers below are backed by the Stage 1F conversion lab (see Stage 1F record).
Short answer: qemu-img changes the container: VMDK to raw or qcow2, sizes, backing chains. It never looks inside the guest. virt-v2v converts the guest: it opens the OS and makes it bootable on KVM (virtio drivers, boot configuration, VMware-specific pieces), then writes the disks and target metadata. So qemu-img alone is not a migration tool.
Detail: In the lab, qemu-img info sees the source disk as vmdk, create type: vmfs, one raw extent, 40 GiB, no backing file. It knows nothing about the netplan file keyed on ens192, which would leave a converted VM without network. virt-v2v 2.4.0 reads a VMware .vmx + .vmdk (-i vmx) or a bare image (-i disk). It can write -o local, -o qemu, -o libvirt or an experimental -o kubevirt (disks plus a YAML file). MTV/Forklift runs virt-v2v in a conversion pod and imports the result with CDI.
Short answer: With libguestfs. It boots its own tiny appliance VM with the disk attached read-only, uses real Linux drivers to mount the guest filesystems, and reads files. The guest OS never runs, so nothing inside it changes.
Detail: On the working copy, virt-inspector identified Ubuntu 24.04.5, hostname legacy-source-vm and 500 packages. virt-cat read fstab, netplan and the nginx site. guestfish --ro -i mounted the root filesystem with ST_RDONLY set. Partition probing (fdisk -l, sfdisk --json, blkid -p with byte offsets) gives the host-level view (GPT, partition GUIDs, filesystem UUIDs), but it cannot read files. The image hash was identical before and after every step.
Short answer: Keep a golden copy that is only ever read, and work on a hash-verified working copy. Lock the working copy, re-hash it after each step, and never attach an image to a running VM while tools inspect it.
Detail: In the lab, the golden copy on Windows was only hashed and copied: its sha256, size, timestamps and attributes were identical before and after. The working copy on the conversion host matched the golden hash on arrival and after re-sparsifying. It was then set to mode 0444 plus chattr +i, so even root’s append failed. qemu-img ran without -U (force-share) and check without -r (repair). Any future conversion writes to a new path.
Short answer: VMDK is the source format, because the VM comes from VMware. Raw is the plain sector image: KubeVirt/CDI store VM disks as raw on a PVC, and an ESXi flat extent is already raw inside. qcow2 is QEMU’s thin, feature-rich format, common for transport, container disks and virt-v2v output (-of qcow2). Which one to use at each step is a design decision, not a default.
Detail: qemu-img probes the lab’s -flat.vmdk on its own as raw: the VMDK wrapper is just the 541-byte descriptor. The target format for KubeVirt was deliberately not chosen in Stage 1F. It depends on CDI’s import path, sparseness during transfer and the conversion method.
Short answer: No. libguestfs falls back to software emulation (TCG) when KVM is unavailable, so a successful run proves nothing about acceleration. Check which accelerator actually ran.
Detail: In the lab, libguestfs requested accel=kvm:tcg. The appliance kernel logged “Hypervisor detected: KVM” and used kvm-clock, and the qemu process held /dev/kvm open: KVM was used. Yet with force_tcg the same self-test finished in 19 s instead of 34 s. The appliance is an L3 guest, so every VM exit crosses two outer hypervisors. “KVM works”, “libguestfs works” and “KVM is faster” are three separate claims.
Short answer: The guest OS owns the filesystems (ext4, vfat, fstab, netplan). The disk container owns the packaging of the bytes (VMDK descriptor + extent, or raw, or qcow2). The hypervisor’s VM metadata owns the virtual hardware and firmware state (.vmx, .nvram). Each tool works at one layer.
Detail: libguestfs works at the guest layer, qemu-img at the container layer, and virt-v2v spans all three: -i vmx reads the .vmx, it converts the container, and it modifies the guest. CDI works at the container layer on the Kubernetes side. The .nvram is never migrated: the target boots with a fresh variable store through the ESP fallback loader (EFI/BOOT/BOOTX64.EFI), which the lab disk has.
Answers below are backed by the Stage 1G experiments (see Stage 1G record).
Short answer: It may boot, but “boots” is not “works”. qemu-img changes only the container, so every VMware assumption inside the guest survives. In the lab the guest booted fine on virtio and served nginx locally, but it had no network.
Detail: qemu-img convert -f vmdk -O qcow2 took 5 s, and qemu-img compare reported the images identical. Ubuntu 24.04 has virtio built into the kernel and mounts by UUID, so storage just worked (/dev/vda). The NIC appeared as enp0s3, but netplan still matched ens192, so it stayed unmanaged and DOWN. open-vm-tools was still installed, just skipped by systemd’s ConditionVirtualization=vmware. A guest that uses /dev/sdX in fstab, or whose initramfs lacks the virtio storage driver, would likely have failed earlier (not tested).
Short answer: Measure it instead of assuming. In the lab it removed VMware Tools, rebuilt the initramfs, added a virtio alias and a first-boot job to install qemu-guest-agent, and generated target metadata. It did not fix the network configuration, regenerate GRUB configuration, or change identity.
Detail: virt-diff between input and output (396 lines) plus the -v -x log showed: dpkg --purge open-vm-tools; bochs added to /etc/initramfs-tools/modules and update-initramfs (old image kept as .pre-v2v); /etc/modprobe.d/virt-v2v-added.conf with alias scsi_hostadapter virtio_blk; a guestfs-firstboot service. It warned that it “could not determine a way to update the configuration of Grub2”. netplan, fstab, grub.cfg, the ESP, machine-id, SSH host keys and nginx files were checksum-identical to the source. The input working copy was unchanged (read-only and immutable throughout).
Short answer: The configuration was keyed to the old hypervisor’s interface name. Fix it by matching the interface by something that survives the move (driver, or a deliberate MAC policy), keep the guest’s own addressing model, and test on an isolated network with a test address, never the production IP while the source is alive.
Detail: netplan generated [Match] Name=ens192; the virtio NIC was enp0s3. On a second disposable copy, one file changed: match: driver: virtio_net, 10.0.2.15/24, gateway 10.0.2.2, DNS 10.0.2.3, dhcp4: false (the original was static too). virt-diff confirmed a single-file change. After boot, the link was routable with a default route, and curl through a QEMU port forward returned HTTP 200 with the source page hash. The same guest identity (machine-id, host keys, hostname, IP) must never appear twice on a real network.
Short answer: Collect evidence on both sides. Host side: the QEMU process holds /dev/kvm, a kvm-vm fd and one kvm-vcpu fd per vCPU, and the monitor says kvm support: enabled. Guest side: systemd-detect-virt says kvm, dmesg shows “Hypervisor detected: KVM”, and lspci shows virtio devices bound to their drivers.
Detail: All three lab boots showed virtio-blk 1af4:1001 (disk vda) and virtio-net 1af4:1000 (virtio_net driver). QEMU was run directly, without libvirt, so the exact command line is the record. A successful boot alone proves nothing about acceleration: QEMU can fall back to TCG if KVM is not requested or not available.
Short answer: Not for this guest. With an empty variable store, OVMF booted the disk through the ESP fallback loader (\EFI\BOOT\BOOTX64.EFI, which is shim), and a Ubuntu boot entry appeared afterwards. Match the Secure Boot setting deliberately rather than by accident.
Detail: The source runs UEFI with Secure Boot off, so the lab used the non-Secure-Boot OVMF_CODE_4M.fd and a fresh OVMF_VARS_4M.fd copy per boot. Switching to BIOS would have been a different test, not an equivalent one. Secure Boot on (signed shim and GRUB are present) is untested. On KubeVirt this translates to EFI boot with an explicit Secure Boot choice.
Short answer: The disk and boot path are the easy part. The hard parts are network identity, guest-agent provisioning and VM metadata. CDI moves bytes; it does not fix any of these.
Detail: virtio storage and UEFI worked without changes. The network needs a remediation designed together with the KubeVirt network binding: a pod network typically hands out an address by DHCP, while a bridged network could keep the static address. virt-v2v’s first boot tries to install qemu-guest-agent from the internet, which stalls in an isolated cluster, so pre-install the agent or provide a mirror. -i disk guessed 1 vCPU / 2 GiB, so VM sizing must come from the .vmx. The target disk format (raw or qcow2 into CDI) remains an explicit decision.
Answers below are backed by the Stage 1H design and its sources (see Stage 1H record). Nothing was provisioned, so these are design answers, not runtime results.
Note added 2026-10-15: Stage 1I later built this design and ran the migration (2026-10-01). The answers below are kept as design answers; runtime answers are in Stage 1I additions.
Short answer: Decide from KubeVirt’s host requirements, not from “managed is easier” or “self-managed is more flexible”. For the first migration I chose a single-node kubeadm cluster on one nested-virtualization EC2 instance and deferred EKS; EKS was not blocked.
Detail: KubeVirt needs /dev/kvm on the worker, a privileged-capable cluster, a supported runtime, and recommends a host kernel from the same family as the virt-launcher userland, which is CentOS Stream 9. EKS offers Kubernetes 1.36 and allows privileged pods, and the CPU option for nested virtualization is not on the managed node group’s prohibited list, but its node images are not EL-based and the /dev/kvm path on EKS nodes is undocumented. Self-managed lets the node run CentOS Stream 9 with CRI-O 1.36, the exact combination in KubeVirt’s own CI provider for Kubernetes 1.36. EKS would also add USD 0.10 per hour (about 42%) to a lab that runs for a few hours. See ADR 007.
Short answer: Not any more for functional work. Some current EC2 families (for example m8i, m7i) can expose VT-x through a nested-virtualization CPU option, which is much cheaper than metal. Metal remains the fallback if nested KVM does not work in the region.
Detail: m8i.xlarge costs about USD 0.22 per hour in ap-south-1; the smallest suitable metal size costs about USD 5 per hour. Nested virtualization costs performance, so it suits a functional migration test, not benchmarks. The decision is only accepted once virt-host-validate qemu passes on the real node; availability of the option in ap-south-1 is still an unknown until checked with the AWS API.
Short answer: Not in this design, and it should not try. 192.168.50.31 lives on a laptop NAT network with no route to AWS, a VPC only delivers addresses the ENI owns, the masquerade binding assigns the guest address, and the source VM still uses .31. What must continue is the service, not the address.
Detail: The guest gets 10.0.2.2 by DHCP inside its virt-launcher pod. Traffic is NATed three times: by masquerade in the pod, by flannel on the node and by the internet gateway. Clients reach nginx through a NodePort Service on port 30080, which stays stable across VM restarts. Continuity is proven by content: HTTP 200, the 267-byte page and its sha256, plus unchanged filesystem UUIDs and configuration hashes. Keeping an IP would need an L2 or routed network that owns that subnet, which is a separate design problem. See ADR 008.
Short answer: Masquerade is the documented pod-network binding that works with any CNI, needs no extra components, and matches what the guest does after remediation (DHCP). Bridge on the pod network forbids live migration and some CNIs dislike the moved MAC; Multus only helps if there is a real secondary L2 network, which a VPC does not provide for guest-chosen addresses.
Detail: passt is the other candidate: it supports live migration, but it needs a Beta feature gate and more memory per VM, and live migration is out of scope for a single node with RWO storage anyway. Multus was not assumed; the chosen network model does not require it.
Short answer: Move qcow2 and let CDI store raw. The qcow2 from virt-v2v is about 2.7 GiB instead of 40 GiB, carries the guest changes, and CDI converts it with qemu-img into a raw 40 GiB Block-mode PVC on EBS gp3.
Detail: The upload goes through virtctl image-upload to cdi-uploadproxy over a kubectl port-forward, so the proxy is never exposed. CDI needs Filesystem scratch space for the conversion, and --force-bind because the StorageClass uses WaitForFirstConsumer. Block mode avoids a filesystem layer under the guest disk. The PVC is ReadWriteOnce, so the VM is not live-migratable; that is accepted and recorded. See ADR 009.
Short answer: Do not install it at first boot. Pre-install qemu-guest-agent offline into a disposable copy of the converted disk on the conversion host, remove virt-v2v’s first-boot apt job, and treat the agent as optional for passing validation.
Detail: In Stage 1G virt-v2v’s first-boot job stalled in apt-get update on an isolated network. The design downloads the package and its dependencies on the conversion host and installs them offline with libguestfs tools. The exact dependency set is still an unknown. The netplan remediation (ADR 010) is applied the same way and in the same pass: match by driver virtio_net, DHCP.
Answers below are backed by the Stage 1I record (2026-10-01). They describe one cold migration of one VM on a temporary single-node target, not a production service.
Short answer: I did a cold copy of the VMDK from ESXi, verified it by sha256, converted it with virt-v2v, and prepared the guest offline. I boot-tested it under QEMU/KVM, moved it over SFTP to a Terraform-built EC2 node running kubeadm, KubeVirt and CDI, uploaded it into a DataVolume, started a KubeVirt VM, and validated nginx end to end. Then I tore everything down.
Detail: Five approval gates: I1 prepare the image, I2 boot-test it, I3 build the AWS platform, I4 migrate, I5 validate and tear down. The target was one m8i.xlarge with nested virtualization in ap-south-1, running CentOS Stream 9, Kubernetes v1.36.5, CRI-O 1.36.6, flannel, KubeVirt v1.9.0, CDI v1.66.1 and the EBS CSI driver. The disk landed raw on a 40 GiB encrypted gp3 Block PVC. The final check from the operator workstation through NodePort 30080 returned HTTP 200 and 267 bytes, with the source page’s sha256. Only the AWS layer is in Terraform; the cluster software was installed step by step, as the record documents. Two design details from answers 54 and 55 were settled at runtime. Immediate binding used the DataVolume annotation cdi.kubevirt.io/storage.bind.immediate.requested rather than --force-bind. The guest agent’s only missing dependency was liburing2.
Short answer: By content and identity, not by IP address.
Detail: The same page hash came back over HTTP. The machine-id, filesystem UUIDs, hostname, nginx package and configuration-tree hash all matched the Stage 1C baseline. The guest’s host key matched the ED25519 key pinned in Stage 1C under strict host-key checking. The things that were expected to change were recorded as such: NIC ens192 became enp1s0, disk sda became virtio vda, the MAC changed, the SMBIOS vendor became KubeVirt, and 192.168.50.31 became 10.0.2.2 by DHCP. DataVolume Succeeded was treated as proof that CDI finished writing, not that the right bytes arrived.
Short answer: Several things, and each one was recorded rather than hidden.
Detail:
restrict=on offers no DHCP router. Nothing was patched; a second run on an isolated network namespace passed..partial file had inherited a read-only mode.rd, which is a built-in alias of Remove-Item. Four were re-collected by re-running the same read-only scripts. The fifth, the post-transfer destination hash check, could not be reproduced, so the record relies on two surviving original records of the same hash and says so.Short answer: In lifecycle order, verifying each layer before destroying the next.
Detail: Delete the VM and Service, and confirm the disk survives because the VM does not own it. Delete the DataVolume; its owner reference removes the PVC, reclaimPolicy: Delete removes the PV, and the CSI driver deletes the EBS volume. Confirm InvalidVolume.NotFound in AWS and check that no migration objects remain in the cluster. Only then review a terraform plan -destroy (exactly 10 resources), apply it, and confirm every resource is gone in AWS. A final detection-only scan for CSI-tagged volumes found nothing. CSI-created volumes are outside Terraform state, which is why this order matters.
Short answer: It proves one cold migration of one Linux VM, with validation and a clean teardown. It does not prove a migration product.
Detail: