kubevirt-vm-migration-platform

ADR 008: Pod network with masquerade binding; the VMware IP is not preserved

• Status: Accepted (Stage 1H, 2026-09-28). Design decision; nothing has been deployed.
• Amended: 2026-09-30, documentation only, from the Stage 1H architecture review:
  • review R3: address plan;
  • review R4: guest management path;
  • clarifications of the address-plan overlap rule and of what the masquerade 10.0.2.0/24 is.

  Nothing was deployed or tested for this amendment.

• Post-implementation (added 2026-10-15): deployed and validated in Stage 1I on 2026-10-01, then removed in the same stage. See Post-implementation validation. The lines above are kept as recorded at the time.

Context

The source VM serves nginx on a static 192.168.50.31 on VMnet8, a VMware Workstation NAT network on the laptop. Stage 1G showed that the converted guest needs a network remediation, and that the remediation depends on how KubeVirt attaches the NIC. KubeVirt v1.9 offers masquerade and bridge bindings on the pod network, bridge on Multus secondary networks, the passtBinding core binding (Beta, on by default in v1.9) and binding plugins [E10, E11, E20]. The target is a single-node kubeadm cluster with flannel on one EC2 instance (ADR 007).

Decision

1. The VM has one interface on the pod network with the masquerade binding, model: virtio, and an explicit port list (80, 22).
   • Port 80 serves the HTTP validation path.
   • Port 22 is declared only for guest management over the Kubernetes API (virtctl ssh or virtctl port-forward), inside KubeVirt’s guest networking model. No Service exposes it, and the AWS security group does not open it (clarified 2026-09-30, review R4).
2. The guest gets its address from KubeVirt’s DHCP: 10.0.2.2/24, gateway 10.0.2.1 (default vmNetworkCIDR 10.0.2.0/24) [E21]. Egress is source-NATed to the pod IP, then to the node’s VPC address.
   • 10.0.2.0/24 is the masquerade binding’s guest-side network inside the VM’s own virt-launcher pod. It is not a VPC subnet and not a network that VMs share (clarified 2026-09-30).
   • Several VMs do not share one L2 segment or compete for one global 10.0.2.2: each can be 10.0.2.2 inside its own pod, and is reached through its pod IP or a Service. See the Stage 1H record, section 13.
3. No Multus and no NetworkAttachmentDefinitions: this model does not need them.
4. 192.168.50.31 is not preserved. Continuity is provided by a Kubernetes Service (NodePort 30080, reachable only from the operator’s /32 for validation). Workload identity is proven by content (page sha256), not by address.
5. NodePort 30080 is external HTTP validation only. It is not a guest management path (added 2026-09-30, review R4). The operator reaches the guest through the Kubernetes API over the node SSH tunnel: virtctl console, virtctl vnc where applicable, and virtctl ssh or virtctl port-forward. See the Stage 1H record, section 29.4.
6. The independent address domains must not overlap (added 2026-09-30, review R3; wording clarified 2026-09-30). The masquerade vmNetworkCIDR stays at its default 10.0.2.0/24. The other ranges are proposed as:
   • VPC 10.40.0.0/16 and subnet 10.40.1.0/24 (proposed design);
   • pod CIDR 10.244.0.0/16 (flannel default);
   • Service CIDR 10.96.0.0/12 (kubeadm default).

   The subnet is intentionally contained within the VPC. The domains that must not overlap one another are the VPC/subnet address space, the pod CIDR, the Service CIDR and the masquerade CIDR, and each of them must also stay clear of CRI-O’s default bridge network 10.85.0.0/16. None of these values is runtime-confirmed. See the Stage 1H record, section 29.3.

Why

Binding	Verdict for the first cold migration
Pod + masquerade	Chosen. Works with any CNI, needs no Multus, is allowed to live-migrate later, and is KubeVirt’s recommended default over pod bridge [E20, E22]
Pod + bridge	Rejected: gives the guest the pod IP anyway, is not allowed to live-migrate, and some CNIs and tools break when the MAC moves [E20, E22]
Multus secondary + bridge	Rejected: needs Multus, NADs and an L2 segment. A VPC only accepts ENI-assigned addresses from the subnet range, so a guest-chosen address is not deliverable [E46]
Pod + passt	Deferred: Beta, 250 Mi extra memory per VM, no benefit here [E10, E20]

The VMware IP cannot survive for four independent reasons: VMnet8 is not reachable from AWS; masquerade assigns the address; a VPC drops foreign addresses; and the source keeps running at .31 as the reference and rollback.

Consequences

• The guest must use DHCP and must not match its NIC by the VMware name. That is ADR 010.
• Clients reach the VM through the Service, never by the guest address; pod IP changes (restart, reschedule) do not matter.
• Infrastructure identity changes (IP 192.168.50.31 -> 10.0.2.2 behind the pod IP, MAC, interface name) are recorded as expected, not as failures.
• Guest DNS is expected to be cluster DNS through the DHCP response (INFERRED; to be observed).
• Deferred: DNS names, LoadBalancer or Ingress, Multus, secondary networks, bridge or passt bindings, MAC preservation, any IP preservation.

Evidence

Stage 1H record, sections 13, 14, 20 and 29, sources E10, E11, E20 to E22, E46, E52 to E54; Stage 1H architecture review, section 8.

Post-implementation validation

Added 2026-10-15. The decision and its rationale above are unchanged. All values are OBSERVED in the Stage 1I record, sections 18 and 19, and in the runtime snapshot (packet path, section 8).

Aspect	Outcome
Design decision	One virtio NIC on the pod network with the masquerade binding, DHCP inside the guest, no Multus, no IP preservation, NodePort 30080 for external HTTP validation only
Deployed and validated	The VirtualMachine manifest declares model: virtio, masquerade: {} and ports 80 and 22. The guest received 10.0.2.2/24 with gateway 10.0.2.1 by DHCP, and DNS 10.96.0.10 (cluster DNS). The consequence above marked “INFERRED; to be observed” is now observed. The NodePort Service on 30080 was reachable only from the operator /32 and returned HTTP 200 with the source page hash. Guest SSH used virtctl port-forward through the Kubernetes API, with no Service and no security-group rule for port 22. The address domains (VPC 10.40.0.0/16, pods 10.244.0.0/16, Services 10.96.0.0/12, masquerade 10.0.2.0/24, CRI-O bridge 10.85.0.0/16) did not overlap
Observed behaviour worth knowing	The guest NIC was named enp1s0 on KubeVirt’s q35 layout, not enp0s3 as in the plain QEMU tests. The driver match (ADR 010) made the name irrelevant. With externalTrafficPolicy: Cluster, nginx saw the client as the cni0 gateway 10.244.0.1, not as the operator workstation
Limitations	One VM and one NIC. No DNS name, LoadBalancer, Ingress, Multus or IP preservation. 192.168.50.31 stayed with the source VM, and no client traffic was cut over
Destroyed	The Service was deleted in I5-C, after which port 30080 refused connections. The security group was destroyed with the rest of the Terraform state in I5-G and confirmed absent in AWS in I5-H