Running a KVM guest on HiKey



For some releases now,  KVM - the virtualization infrastructure that turns the Linux kernel into an hypervisor- has been available in the HiKey from 96Boards.  This was made possible thanks to the open nature of the HiKey software stack but also to the early work done by Marc Zyngier as captured on this post and the numerous contributions to aarch64 from the Linaro Virtualization team.

The second version of the VM System Specification for ARM processors was released back in April 2016: this document aims at [..] providing a set of guidelines for both guest OS and hypervisors such that OS images built according to these guidelines shall guarantee that those images will be able to execute on those hypervisors […].  You can download the 2.0 version of the specification on this link.

Having said that and despite the specification being available - but perhaps due to it being so recent-  you might have some issues finding OS images that comply to it.

However this doesn’t prevent you from running a networking enabled Linux guest on your octa-core HiKey; as a matter of fact, there is a simple yet efficient way to execute a Linux guest OS on any Linux KVM enabled system without the need for more complex - and powerful- software virtualization packages: the native KVM tool, ie kvmtool.

But of course you can still use QEMU if that is what you like.

jro@HiKey:~# qemu-system-aarch64 -machine virt -cpu cortex-a53 -machine type=virt -nographic -smp 1 -m 512 -kernel /boot/vmlinuz-4.4.0-135-arm64 --append "console=ttyAMA0" --enable-kvm

How to Install and Run kvmtool.

Depending on whether your interest leans towards learning about the KVM API or simply in starting another Linux instance as a guest, you might choose to either install from source or  use a package manager. We will discuss building from source in the next section and focus on the quick way of running it on this one. 

To install the kvmtool on Debian run the following on a terminal:

jro@HiKey:~# apt-get install kvmtool

By default,  the HiKey kernel should have been configured with all the recommended settings for kvmtool (see README). If you found that not to be the case,  follow the HiKey building from source instructions and replace the kernel.  

To check for a  particular kernel config in a running kernel, for example CONFIG_NET_9P, do the following:

jro@HiKey:~# zcat /proc/config.gz | grep CONFIG_NET_9P

A good way of enabling configs before re-building a kernel is to use the scripts/config that is present in the Linux tree:

jro@HiKey:~/linux.git# scripts/config --enable CONFIG_NET_9

If you have used kvmtool on other platforms and never had the need to specify a kernel image  be aware that arm64 requires an uncompressed kernel to boot (see section 3 of the booting arm64 kernel document): on arm64 booting the guest with the default options fails since the default kernels handled by kvmtool are compressed images.

This failure will show something like this on a terminal:

jro@HiKey:~# lkvm run

# lkvm run -k /boot/vmlinuz-4.4.0-135-arm64 -m 256 -c 2 --name guest-1715

Info: Loaded kernel to 0x80080000 (15869952 bytes)

Info: Placing fdt at 0x8fe00000 - 0x8fffffff

Info: virtio-mmio.devices=0x200@0x10000:36

Info: virtio-mmio.devices=0x200@0x10200:37

Info: virtio-mmio.devices=0x200@0x10400:38

Fortunately you can kill the process from another terminal with SIGTERM (ie: kill pid)

jro@HiKey:~#  kill `pidof lkvm`

To use kvmtool to start a guest on arm64 you have to use the option ‘-k’ with the uncompressed kernel image: in the case above that failed, you could boot the guest it by gzip decompress the vmlinuz file (/boot/vmlinuz-4.4.0-135-arm64) and pass it to the kvmtool.

jro@HiKey$ lkvm run -k/boot/vmlinuz-4.4.0-135-arm64.uncompressed

Now on to the guest’s filesystem: one of the neat features that kvmtool provides is the automatic creation of a simple root file system derived from that of the host. By default the host shares its filesystem with the guest placing the guest’s in ~/.lkvm/default/.  This is achieved via using virtfs (virtualization aware file system pass-through).

guest# zcat config.gz | grep 9P




guest# mount

/dev/root on / type 9p (rw,relatime,dirsync,trans=virtio,version=9p2000.L,cache=loose)

hostfs on /host type 9p (ro,relatime,sync,dirsync,trans=virtio,version=9p2000.L)

On boot  this is what you should expect to see on the top directory:

guest# ls -la

total 36

drwxr-xr-x 12 root 0 4096 Aug 25 2016 .

drwxr-xr-x 12 root 0 4096 Aug 25 2016 ..

lrwxrwxrwx 1 root 0 9 Aug 25 2016 bin -> /host/bin

drwxr-xr-x 6 root 0 2800 Jan 1 00:00 dev

drwxr-xr-x 2 root 0 4096 Aug 25 2016 etc

drwxr-xr-x 2 root 0 4096 Aug 25 2016 home

drwxr-xr-x 22 root 0 4096 Aug 24 2016 host

lrwxrwxrwx 1 root 0 9 Aug 25 2016 lib -> /host/lib

lrwxrwxrwx 1 root 0 11 Aug 25 2016 lib64 -> /host/lib64

dr-xr-xr-x 56 root 0 0 Jan 1 00:00 proc

drwxr-xr-x 5 root 0 4096 Aug 25 2016 root

lrwxrwxrwx 1 root 0 10 Aug 25 2016 sbin -> /host/sbin

dr-xr-xr-x 12 root 0 0 Jan 1 00:00 sys

drwxr-xr-x 2 root 0 4096 Aug 25 2016 tmp

lrwxrwxrwx 1 root 0 9 Aug 25 2016 usr -> /host/usr

drwxr-xr-x 3 root 0 4096 Aug 25 2016 var

drwxr-xr-x 3 root 0 4096 Aug 25 2016 virt

With respect to networking, the guest  will be assigned the IP address (click on it to browse the include file in the tree); executing the ifconfig command on the guest should return something like this:

guest# ifconfig

eth0 Link encap:Ethernet HWaddr 02:15:15:15:15:15

inet addr: Bcast: Mask:


guest# ping

64 bytes from icmp_seq=3 ttl=64 time=0.928 ms


In summary -and as an example- to boot an arm64 Image using kvmtool and giving the guest two cores, 256MB and a virtual console on the HiKey board you could do:

jro@HiKey$ lkvm run --console serial -c2 -m256 -k Image -n mode=user,trans=mmio -p console="ttyS0 earlycon=uart,mmio,0x3f8"

When you are ready to exit the VM type “exit” on its console:

guest# exit

reboot: Restarting system

# KVM session ended normally.

Building the kvmtool from source and tracing the KVM.

To install the kvmtool from source just clone the project and follow the README or perhaps its INSTALL instructions (preferably the later if you need further details or prefer  to cross-compile it instead of building natively on the HiKey).

jro@HiKey:~# git clone git://

If you are rebuilding the host kernel - maybe to add some extra options -  I’d recommend you to enable  ftrace support:  this will allow you to  capture and analyze the behavior of KVM on the running system. (by default the Linaro Reference Platform Builds enables this option).

To get a list of all the KVM events that you can monitor from your host:

jro@HiKey:~# trace-cmd  list | grep kvm

Tracing KVM (ie, generating the trace.dat file that captures the KVM operation) is as simple as initiating a record session, executing the workload on the guest, stopping the recording session and analyzing the generated data.

At this point and to wrap it all up, you should be able to boot an arm64 Linux guest with network access on an arm64 Linux host using the HiKey board; you should also be able to trace and debug the host KVM functionality (perhaps, having read the kernel’s virtualization time keeping documents  you are still curious about realtime KVM).

Moreover, if you are interested on developing your own KVM based host tool, have a look at how kvmtool does things (there are plenty of documents on the web as well as presentations).

comments powered by Disqus