KVM VGA-Passthrough using the new vfio-vga support in kernel = 3.9

Question

This is intented for people who wish to pass-through a GPU to a virtual machine using the KVM hypervisor, QEMU and vfio-pci
NOTE: AMD RADEON 5xxx, 6xxx, 7xxx, 2xx and NVIDIA GEFORCE 7, 8, 4xx, 5xx, 6xx, 7xx 9xx have been reported working with this, passing though an intel IGD is not supported YET
NOTE: THIS IS EXPERIMENTAL SO IT MIGHT NOT WORK ON YOUR SYSTEM
This is the result on my radeon 6950:
This is the result on my geforce 470 gtx:
Recommended reads
Alex Williamson's blog and FAQ for detailed technical information
OVMF whitepaper.
GPU passthrough database by noctavian
At this moment there are two ways to achieve this using KVM: legacy VGA via Seabios or non-VGA (UEFI) via OVMF
The main practical advantage of using OVMF is that on intel systems you no longer need the i915 VGA arbiter patch which also disables DRI on the host
Requirements
AMD-VI/VT-D enabled and working (On intel systems both your MB and you CPU must support it, to find out if you CPU has VT-D support go here: http://ark.intel.com/, you also need to boot with intel_iommu=on to enable it)
At least 2 GPU's, one primary boot device and the card you wish to pass-through
Qemu=>2.0
Aditional kernel patches might be required if you're using an Intel CPU: ACS override patch and i915 VGA arbiter patch, you can find a kernel package with these patches included on AUR: linux-vfio
Aditional requirements for OVMF:
An UEFI compatible GPU (Most modern gpus support this, SEE: Does my graphics card ROM support EFI?
An UEFI compatible GUEST (ex: Windows 7 and up)
OVMF:  Gerd Hoffmann's rpms or OVMF-svn from AUR
If you're building your kernel manually this option is required:
CONFIG_VFIO_PCI_VGA=y
Preparing your System
On intel system the i915 VGA arbiter patch is required if:
You're using the IGD on your host
You're doing VGA assignment
NOTE: Enabling this will disable DRI!
If you wish to avoid this, then then you should try OVMF
To enable it add this kernel parameter to your bootloader:
i915.enable_hd_vgaarb=1
Next, we need to prepare the GPU for vfio:
Blacklist radeon or nouveau or nvidia or fglrx on /etc/modprobe.d/blacklist.conf
Example, blacklisting the opensource radeon module:
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
Use pci-stub
In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub
NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this
mkinitcpio -p linux-mainline
lspci
07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] <-- radeon 6950
07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] <-- radeon 6950 audio
lspci -n
07:00.0 0300: 1002:6719 <-- radeon 6950
07:00.1 0403: 1002:aa80 <-- radeon 6950 audio
Now add this kernel parameter to your bootloader:
pci-stub.ids=1002:6719,1002:aa80
dmesg | grep pci-stub
[ 2.096151] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.096160] pci-stub 0000:07:00.0: claimed by stub
[ 2.096165] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.096174] pci-stub 0000:07:00.1: claimed by stub
[ 2.096178] pci-stub: add 1B21:1042 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
Setting up vfio and kvm modules
This is only required if you get this message:
vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
If your board doesn't enable interrupt remapping, you need to add this to your bootloader:
vfio_iommu_type1.allow_unsafe_interrupts=1
Or if vfio-pci was built as a module ( default on arch )
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf
Some applications like Passmark Performance Test and SiSoftware Sandra crash the VM without this:
echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf
Binding a device to vfio-pci
We'll use this script to make life easier:
#!/bin/bash
modprobe vfio-pci
for dev in "$@"; do
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done
Save it as /usr/bin/vfio-bind
chmod 755 /usr/bin/vfio-bind
Bind the GPU and GPU audio:
vfio-bind 0000:07:00.0 0000:07:00.1
Systemd service:
[Unit]
Description=Binds devices to vfio-pci
After=syslog.target
[Service]
EnvironmentFile=-/etc/vfio-pci.cfg
Type=oneshot
RemainAfterExit=yes
ExecStart=-/usr/bin/vfio-bind $DEVICES
[Install]
WantedBy=multi-user.target
cat /etc/vfio-pci.cfg
DEVICES="0000:00:11.0 0000:04:00.0 0000:05:00.0 0000:06:00.0 0000:07:00.0 000:07:00.1"
Testing if its working out
Seabios:
qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-device vfio-pci,host=07:00.0,x-vga=on -device vfio-pci,host=07:00.1 \
-vga none
OVMF:
qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off  \
-smp 4,sockets=1,cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
-device vfio-pci,host=07:00.0 -device vfio-pci,host=07:00.1 \
-vga none
Note:
kvm=off will hide the kvm hypervisor signature, this is required for NVIDIA cards, since its driver will refuse to work on an hypervisor and result in Code 43 on windows (unless you're using a QUADRO)
x-vga=on is required for vga assignment
-vga none disables the default vga device on QEMU, it is also required for vga assignment
You should see a black qemu window on your main display, and seabios/ovmf ouput on your monitor from your passthru'd card saying it cant find anything to boot
If you're using an intel cpu and nothing happens try this:
modprobe -r kvm_intel
modprobe kvm_intel emulate_invalid_guest_state=0
NOTE: There might be some problems using nvidia/fglrx/nouveau drivers on the host gpu, see the ISSUES section below on how to solve this
At this point you're ready to create your vm, you could use libvirt + virt-manager to create your vm using a nice graphical interface, there are
also some interesting examples in this thread, otherwise if you prefer to use the command line continue reading
DISK:
Creating a disk image:
Create a 30Gb raw image:
dd if=/dev/zero of=windows.img bs=1M seek=30000 count=0
Using a VIRTIO controller:
We create our virtio controller for qemu:
-device virtio-scsi-pci,id=scsi
Attaching our hdd:
-drive file=/home/nbhs/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk
Attaching a cdrom:
-drive file=/home/nbhs/windows.iso,id=isocd,if=none -device scsi-cd,drive=isocd
To install windows, you'll need to download the virtio drivers iso from here
Attaching the virtio iso:
-drive file=/home/nbhs/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd
Windows will complain it cant find a disk to install to, just click browse, find the folder for your os/arch, select "RED HAT VirtIO pass-through controller" then click next.
For more info check out the Archlinux wiki's QEMU section
Using a physical disk or partition:
-drive file=/dev/sdb,id=disk,format=raw,if=none -device scsi-hd,drive=disk
You might need to change the bus depending on the controller you use
If you wish to use a physical partition and be able to read its contents later on, you can follow these guides:
http://fds-team.de/cms/articles/2013-12 … yer-u.html
https://wiki.archlinux.org/index.php/QE … disk_image
Passing a sata disk controller to qemu:
If you dual boot like me you can pass an entire sata controller (in ahci mode) and all the drives attached, and seabios will boot from them.
lspci
00:11.0 SATA controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
Make sure all volumes are unmounted, then bind the controller to vfio:
vfio-bind 0000:00:11.0
Now we pass the controller to the vm:
-device vfio-pci,host=00:11.0
USB:
Its also possible to passthrough a usb controller, in my case an ASMEDIA USB3 Controller:
lspci
04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
Bind them to vfio:
vfio-bind 0000:04:00.0 0000:05:00.0 0000:06:00.0
Pass them to the vm:
-device vfio-pci,host=04:00.0 \
-device vfio-pci,host=05:00.0 \
-device vfio-pci,host=06:00.0
Or a specific device:
lsusb
Bus 004 Device 010: ID 045e:00e1 Microsoft Corp. Wireless Laser Mouse 6000 Reciever <-- mouse
Bus 004 Device 011: ID 045e:074b Microsoft Corp. <-- keyboard
We pass them to the vm:
-usb -usbdevice host:045e:00e1 -usbdevice host:045e:074b
NETWORK:
Please see https://wiki.archlinux.org/index.php/QEMU#Networking
AUDIO EMULATION:
To hear the sound from the vm on your host speakers you'll need to add this lines:
-soundhw hda
You might need to start qemu like this:
QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...
alsa:
QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64...
Note: these are the settings i found that work great on my system, if you get crackling/skipping audio you might want to try different settings
To see the available drivers and audio options:
qemu-system-x86_64 -audio-help
LOADING YOUR CARD ROM FROM A DUMP/FILE:
modify this line:
-device vfio-pci,host=07:00.0,...... \
change it to this:
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \
ISSUES:
Using the latest NVIDIA drivers on the guest will result in code 43, but there's a workarround
Using the latest NVIDIA drivers on the guest using hv enlightenments will result in code 43, see: https://forums.geforce.com/default/topi … 8/#4314318
Using AMD proprieraty drivers on the host, see: https://bbs.archlinux.org/viewtopic.php … 2#p1273412 the only solution atm is using the opensource radeon driver
PERFORMANCE IMPROVEMENTS:
SEE: https://bbs.archlinux.org/viewtopic.php … 1#p1270311
Pin vcpu to pcpu, see http://www.linux-kvm.com/content/tip-ru … cific-cpus
Use cgroups, specifically cpuset, see https://access.redhat.com/site/document … puset.html
Use Hugepages, see http://pic.dhe.ibm.com/infocenter/lnxin … ttunhp.htm
Use Hyper-V enlightenments, see: http://blog.wikichoon.com/2014/07/enabl … h-kvm.html
Last edited by nbhs (2015-04-05 20:01:18)

Answer 1

nbhs wrote:
apoapo wrote:
I finally got some progress.
After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things .
NIce! have you tried running windows with it yet? does it work if you dont load the rom file by hand?
Hi,
I tried it again and I captured this error message if I use the PCI-e card es Primary Card:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:01:00.0 BAR 0 mmap unsupported. Performance may be slow
This was, when I got some seabios picture.
After some reboots I wanted to try it again, I was spammed with errors . Dont have them to show you now...  will look for it.
Now I'm With the onboard as primary Card and I found this in dmesg:
[ 0.401391] DMAR: No ATSR found
[ 0.401412] IOMMU 0 0xfed90000: using Queued invalidation
[ 0.401413] IOMMU 1 0xfed91000: using Queued invalidation
[ 0.401415] IOMMU: Setting RMRR:
[ 0.401422] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[ 0.402590] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbd568000 - 0xbd59afff]
[ 0.402606] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbd568000 - 0xbd59afff]
[ 0.402619] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd568000 - 0xbd59afff]
[ 0.402628] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 0.402635] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.402717] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[ 0.402782] ------------[ cut here ]------------
[ 0.402786] WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x83/0x90()
[ 0.402787] Hardware name: To Be Filled By O.E.M.
[ 0.402788] Modules linked in:
[ 0.402790] Pid: 1, comm: swapper/0 Not tainted 3.9.2-1-mainline #1
[ 0.402791] Call Trace:
[ 0.402796] [<ffffffff81058090>] warn_slowpath_common+0x70/0xa0
[ 0.402797] [<ffffffff8105817a>] warn_slowpath_null+0x1a/0x20
[ 0.402799] [<ffffffff8129ea43>] pci_find_upstream_pcie_bridge+0x83/0x90
[ 0.402802] [<ffffffff813b3c7b>] intel_iommu_add_device+0x4b/0x1f0
[ 0.402805] [<ffffffff813ab630>] ? bus_set_iommu+0x50/0x50
[ 0.402806] [<ffffffff813ab65a>] add_iommu_group+0x2a/0x50
[ 0.402809] [<ffffffff8135bf63>] bus_for_each_dev+0x63/0xa0
[ 0.402811] [<ffffffff813ab622>] bus_set_iommu+0x42/0x50
[ 0.402814] [<ffffffff819130b6>] intel_iommu_init+0xa83/0xb7b
[ 0.402816] [<ffffffff818dd10b>] ? memblock_find_dma_reserve+0x124/0x124
[ 0.402818] [<ffffffff818dd11d>] pci_iommu_init+0x12/0x3c
[ 0.402820] [<ffffffff8100210a>] do_one_initcall+0x10a/0x160
[ 0.402823] [<ffffffff818d5037>] kernel_init_freeable+0x15b/0x1dc
[ 0.402824] [<ffffffff818d4881>] ? do_early_param+0x88/0x88
[ 0.402827] [<ffffffff814b55f0>] ? rest_init+0x90/0x90
[ 0.402828] [<ffffffff814b55fe>] kernel_init+0xe/0x190
[ 0.402830] [<ffffffff814dad6c>] ret_from_fork+0x7c/0xb0
[ 0.402832] [<ffffffff814b55f0>] ? rest_init+0x90/0x9