Fast cloning of a libvirt/KVM Virtual Machine

We are currently in the process of migrating our VM-based builders from VirtualBox to libvirt/KVM. Our target VMs are some Windows10 hosts that have a 100GB harddisk. Using virt-clone to duplicate our master VM for the actual build process needs to copy the entire disk, which takes much too long, even on decent hardware.

Of course, our VMs are only ephemeral: they are created to run a build-process (possibly downloading and installing stuff). Once the build-process has finished (and all artifacts have been recovered), the VM is then destroyed. So, the full disk clone will only be short-lived.

Most of the data is already present on the master disk image (so we could re-use that), but we must make sure that whatever data we create, does not taint the master image.

Some clever people have solved this problem in the past, with a technique named copy on write (COW).

The basic idea is, that a master disk image is an immutable storage that cannot be modified by the cloned VMs. Instead, whenever a cloned VM needs to modify the storage, only the difference data is written. For long-running VMs that modify a lot of the original system, this can become a performance issue (as the difference keeps growing, and accessing the storage means that the differences have to be resolved). Luckily, our use-case is short-lived VMs where the difference data should stay reasonably small.

Unfortunately, the virt-clone tool always does full clones of the storage media, or no clones at all. (Actually there's also the --reflink flag for COW copies, but this only works with btrfs. Presumably some people are using btrfs, but we are not among them.)

So we need to build our own variant: virt-clonefast.

Example VM

For the sake of this blog, I'm using a VM with the following setup:

QCOW2

Since our hypervisor uses libvirt/KVM, the obvious COW-format is qcow2 ("QEMU Copy On Write", version 2).

If our master image is already in the qcow2 format, it's easy to create a new image based on the original:

1qemu-img create     \
2 -b debian12.qcow2  \ # backing file
3 -F qcow2           \ # backing format
4 -f qcow2           \ # output format
5 debian12-clone.qcow2 # output file

This will create a new disk image file debian12-clone.qcow2, using the qcow2 format (-f). This disk image is based on the debian12.qcow2 backing file (-b), which also uses the qcow2 format (-F).

Checking the sizes of the two image files, we see that while debian12.qcow2 is large, debian12-clone.qcow2 is very small:

Also the clone process took less than a second.

Manually cloning a VM without disks

Per default virt-clone clones all the disk data, making it rather slow:

1time virt-clone -o debian12 -n debian12-clone --auto-clone
2Allocating 'debian12-clone.qcow2'                                                                                                                                                                                     |    0 B  00:00:00 ... 
3
4Clone 'debian12-clone' created successfully.
5
6real    0m0.969s
7user    0m0.116s
8sys     0m0.027s

Well, obviously it's fast enough for the 5GB disk image (which really only has 1.5GB of actual data). Things do change once we are cloning a 100GB disk image with 60GB of data, where it can take 10 minutes or so....

Anyhow: Luckily, we can tell it to skip cloning a given disk with the --skip-copy parameter.

We first need to detect which disks we want to not-copy:

1$ virsh domblklist debian12
2 Target   Source
3--------------------------------------------------
4 vda      /var/lib/libvirt/images/debian12.qcow2
5 sda      -

As we can see, we have a vda disk (powered by our disk image) and an sda disk (which is the CD-ROM).

We do not want to automatically clone the disks, so we omit the --auto-clone option. As of virt-clone 4.1.0, we must either provide the --auto-clone option or specify the storage backend for the various disks manually with the --file option (once per each harddisk).

Therefore, a manual clone of the VM omitting the full clone of the large disk image looks like this:

1virt-clone -o debian12 -n debian12-shallow --skip-copy=vda --file=/var/lib/libvirt/images/debian12-shallow.qcow2

The clone process was very fast (as no copying was involved), but for whatever reasons, virt-clone ignored the --file parameter, and the resulting VM now uses the same disk image as our source VM.

We do not want this. It seems we need to manually detach the old disk image, and then manually attach the new (shallow) copy:

The easiest way, is to do that via the virt-manager: 0. Power down the VM (if not already shut off)

1. Double click the VM
2. Go the the VM details
3. Locate the VirtIO Disk 1
4. Remove it
5. Click on Add Hardware
6. Select Storage (which is the default)
7. Add the bottom of the options select Disk device (for device) and VirtIO (for bus)
8. Select the Select or create a custom storage option
9. Click the Manage... button
   • This shows all the available storage pools. locate the one containing the shallow disk image.
   • If the disk image is in a directory that is not yet part of a pool, create a new pool of type dir
   • If the disk image does not show up in the relevant pool, you probably have to Refresh the volume list first (clicking on the Reload button)
10. Select the disk image
11. Add the disk by clicking the Finish button.

Now you can boot the cloned VM with the shallow harddisk image.

Scripting (1st attempt)

While the actual cloning of the VM and the disk image can be easily scripted (see above), the re-attaching of the shallow disk clone involves a lot of clicking and is not usable for an automated workflow.

My somewhat naive solution was to use libvirt's detach-disk resp attach-disk.

Since this is our system disk, we need to detach/attach with the VM powered off. otoh, the detach-disk and attach-disk commands are meant for hot-plugging devices (which requires the VM to be running). In order to force the operation with the powered-off VM, we need to add the --current flag (which skips the is-running check). Alternatively, you could use the --persistent flag (which enforces the VM to be powered off).

So detach the disk:

1virsh detach-disk             \
2    --domain debian12-shallow \
3    --current                 \
4    --target vda

And re-attach it:

1virsh attach-disk             \
2    --domain debian12-shallow \
3    --current                 \
4    --target vda              \
5    --targetbus virtio        \
6    --source /var/lib/libvirt/images/debian12-shallow.qcow2

Everything looks good, until we boot:

Scripting (fix)

Carefully comparing the XML VM definitions (obtained with virsh dumpxml) between the working one (created with virt-manager) and the broken one (manually using attach-disk), we notice the following:

This can easily be fixed by specifying the --subdriver option when attaching the disk (and in order to kee

1virsh attach-disk             \
2    --domain debian12-shallow \
3    --current                 \
4    --subdriver qcow2         \ # make sure to use QCOW2 features!
5    --target vda              \
6    --targetbus virtio        \
7    --source /var/lib/libvirt/images/debian12-shallow.qcow2

Everything in a single script

With the above knowledge, we can put everything into a single shell script virt-clonefast that clones a given VM with COW-semantics.

It is called like virt-clonefast oldVM newVM to clone the existing oldVM VM into a new VM newVM. QCOW2 disks of the source VM are converted to shallow COW in the same directory as the reference images, and with -shallow# specifier in the name.

Not much error checking is done. Little cleanup is done is case of failure.

 1#!/bin/sh
 2src="$1"
 3dst="$2"
 4
 5
 6usage() {
 7    cat >/dev/stderr <<EOF
 8usage: $0 <srcdomain> <targetdomain>
 9    clone <srcdomain> into <targetdomain> (with shallow copy)
10EOF
11    exit 1
12}
13
14get_qcow2() {
15    virsh domblklist "$1" | grep "\s/.*qcow2$" | while read -r dev path; do
16	test ! -f "${path}" || echo "$dev $path"
17    done
18}
19
20# check arguments
21test -n "${src}" || usage
22test -n "${dst}" || usage
23virsh domstate "${src}" > /dev/null 2>&1 || usage
24virsh domstate "${dst}" > /dev/null 2>&1 && usage
25
26# a place to put out temporary files
27tmpdir=$(mktemp -d)
28_cleanup_tmpdir() {
29    test ! -d "${tmpdir}" || rm -rfv "${tmpdir}"
30}
31trap '_cleanup_tmpdir' EXIT INT TERM
32
33
34# create shallow copies of all qcow2 volumes
35virsh domblklist "${src}" | grep "\s/.*qcow2$" | while read -r dev infile; do
36    test -f "${infile}" || continue
37    basename=${infile%.*}
38    ext=.${infile##*.}
39    infix=0
40    while true; do
41        infix=$((infix+1))
42        outfile="${basename}-shallow${infix}${ext}"
43		touch "${tmpdir}/img"
44        mv -n "${tmpdir}/img" "${outfile}" || continue
45        break
46    done
47    qemu-img create \
48         -F qcow2 -b "${infile}" \
49         -f qcow2 "${outfile}"
50    echo "${dev} ${outfile}" >> "${tmpdir}/disks.txt"
51done
52
53# clone the VM (but skip the shallow disks)
54virt-clone -o "${src}" -n "${dst}" $(cat "${tmpdir}/disks.txt" | while read -r dev file; do echo --skip-copy ${dev} --file ${file}; done)
55
56# attach the shallow disks instead of their full copies
57cat "${tmpdir}/disks.txt" | while read -r dev file; do
58    virsh detach-disk --domain "${dst}" --current --target "${dev}"
59    virsh attach-disk --domain "${dst}" --current --target "${dev}" \
60	  --targetbus virtio --subdriver qcow2 --source "${file}"

Caveats

the above script is not very well tested.

• it is known to not properly handle whitespace in the path to the image disks
• it blindly assumes that all QCOW2 images have the .qcow2 extension
• it does not check whether a file with a .qcow2 extension is indeed a QCOW2 image
• it does not handle images of other formats
• multiple disks are untested

all of these issues are likely to fail catastrophically. be warned.