Berik Ashimov
Senior IT Engineer

GPU / PCI Passthrough: The Path That Works (and What Breaks It)

· 8 min read
#proxmox #passthrough #virtualization

GPU passthrough lets a VM directly access a physical GPU. No emulation, no virtualization overhead — the VM sees real hardware and gets real performance. It’s the only way to run GPU workloads (gaming, machine learning, transcoding) in VMs without massive performance loss.

It’s also one of the most finicky things to configure. Hardware compatibility, IOMMU groups, driver issues — any of these can break passthrough completely. When it works, it’s magical. When it doesn’t, debugging is painful.

Passthrough is hardware compatibility plus attention to detail.

Prerequisites

Hardware Requirements

CPU:

  • Intel: VT-d (IOMMU support)
  • AMD: AMD-Vi (IOMMU support)

Check BIOS for “VT-d,” “AMD-Vi,” “IOMMU,” or “Virtualization Technology for Directed I/O.”

Motherboard:

  • Must support IOMMU
  • Consumer boards often have poor IOMMU groups
  • Server/workstation boards usually work better

GPU:

  • Most discrete GPUs work
  • NVIDIA consumer cards have “Code 43” issues (we’ll address)
  • AMD cards generally work well
  • Intel integrated graphics: limited support

Check IOMMU Support

Terminal window
# Intel
dmesg | grep -e DMAR -e IOMMU


# AMD
dmesg | grep AMD-Vi


# Should see messages like:
# DMAR: IOMMU enabled
# AMD-Vi: Enabling IOMMU

If no messages, enable IOMMU in BIOS.

Enable IOMMU

Edit GRUB Configuration

Terminal window
# Edit GRUB config
nano /etc/default/grub

For Intel:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

For AMD:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Apply changes:

Terminal window
update-grub
reboot

Verify IOMMU Enabled

Terminal window
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi


# Should see:
# DMAR: IOMMU enabled
# or
# AMD-Vi: AMD IOMMUv2 loaded

IOMMU Groups

IOMMU groups are sets of devices that must be passed through together. You can’t pass a single device if it’s in a group with other devices.

View IOMMU Groups

#!/bin/bash
# Save as /root/iommu-groups.sh
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done
done

Example output:

IOMMU Group 1:
    00:01.0 PCI bridge [0604]: Intel Corporation...
    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation... [10de:2204]
    01:00.1 Audio device [0403]: NVIDIA Corporation... [10de:1aef]

The GPU (01:00.0) and its audio device (01:00.1) are in the same group. You must pass both.

ACS Override (If Needed)

Poor IOMMU grouping (everything in one group) can be fixed with ACS override patch. Use with caution — it reduces isolation security.

Terminal window
# Add to GRUB
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"


update-grub
reboot

After reboot, check groups again. They should be smaller.

VFIO Configuration

VFIO (Virtual Function I/O) binds devices for passthrough.

Identify Device IDs

Terminal window
lspci -nn | grep -i nvidia
# 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204]
# 01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef]

Device IDs: 10de:2204 (GPU), 10de:1aef (Audio)

Configure VFIO

Terminal window
# Add VFIO modules
echo "vfio" >> /etc/modules
echo "vfio_iommu_type1" >> /etc/modules
echo "vfio_pci" >> /etc/modules
echo "vfio_virqfd" >> /etc/modules


# Bind devices to VFIO (use YOUR device IDs)
echo "options vfio-pci ids=10de:2204,10de:1aef disable_vga=1" > /etc/modprobe.d/vfio.conf


# Blacklist host drivers (so host doesn't grab GPU)
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia_drm" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidiafb" >> /etc/modprobe.d/blacklist.conf


# For AMD
echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf


# Update initramfs
update-initramfs -u


# Reboot
reboot

Verify VFIO Binding

Terminal window
lspci -nnk -s 01:00
# Should show:
# Kernel driver in use: vfio-pci

If it shows nvidia or nouveau, the blacklist didn’t work. Check modprobe configuration.

Create VM with GPU Passthrough

VM Configuration

Terminal window
# Create VM
qm create 100 --name gpu-vm --memory 16384 --cores 8 --sockets 1 \
  --bios ovmf --machine q35 \
  --net0 virtio,bridge=vmbr0 \
  --scsihw virtio-scsi-pci


# Add EFI disk
qm set 100 --efidisk0 local-zfs:1,format=raw


# Add main disk
qm set 100 --scsi0 local-zfs:100,ssd=1


# CPU settings (important for passthrough)
qm set 100 --cpu host,hidden=1,flags=+pcid


# Add PCI devices
qm set 100 --hostpci0 01:00,pcie=1,x-vga=1


# Add audio device if in same IOMMU group
qm set 100 --hostpci1 01:00.1,pcie=1

Important Settings

BIOS: OVMF (UEFI)

  • Required for modern GPUs
  • Enables PCI passthrough features

Machine: q35

  • Modern chipset with proper PCIe support

CPU: host,hidden=1

  • host: Pass through all CPU features
  • hidden=1: Hide hypervisor from VM (needed for NVIDIA)

hostpci: pcie=1,x-vga=1

  • pcie=1: Use PCIe mode (required)
  • x-vga=1: Primary graphics device

NVIDIA-Specific Fixes

NVIDIA drivers detect virtualization and refuse to work (“Error 43”). Several workarounds:

Hide Hypervisor

Already done with cpu: host,hidden=1. For older NVIDIA drivers, you may also need:

Terminal window
# In /etc/pve/qemu-server/100.conf, add to args (if Error 43 persists):
args: -cpu 'host,hv_vendor_id=NV43FIX,+kvm_pv_unhalt,+kvm_pv_eoi'

Note: Modern NVIDIA drivers (535+) usually work with just hidden=1.

Vendor ID Spoofing

Terminal window
# Add to VM config
qm set 100 --args "-cpu 'host,hv_vendor_id=randomid'"

ROM File (Sometimes Needed)

Some GPUs need their VBIOS dumped and passed:

Terminal window
# Dump ROM (from another system or Windows)
# Or download from TechPowerUp


# Add to VM config
qm set 100 --hostpci0 01:00,pcie=1,x-vga=1,romfile=gpu.rom

Place ROM file in /usr/share/kvm/.

AMD GPU Passthrough

AMD is generally easier:

Terminal window
# VFIO config (use AMD device IDs)
echo "options vfio-pci ids=1002:xxxx,1002:xxxx" > /etc/modprobe.d/vfio.conf


# Blacklist
echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf


update-initramfs -u
reboot

AMD GPUs usually work without additional tweaks.

AMD Reset Bug

Some AMD GPUs (Polaris, Navi) have reset bugs — VM shutdown leaves GPU in bad state, requiring host reboot.

Workaround:

Terminal window
# Install build dependencies
apt install pve-headers-$(uname -r) git build-essential dkms


# Clone and build vendor-reset module
git clone https://github.com/gnif/vendor-reset.git
cd vendor-reset
dkms install .


# Verify module loads
modprobe vendor-reset


# Make persistent after successful test
echo "vendor-reset" >> /etc/modules

Other PCI Devices

Passthrough works for any PCI device, not just GPUs:

USB Controller

Pass entire USB controller for low-latency USB:

Terminal window
# Find USB controller
lspci | grep USB


# Pass through
qm set 100 --hostpci2 00:14.0,pcie=1

NVMe Controller

Pass NVMe for direct storage access:

Terminal window
# Find NVMe
lspci | grep NVMe


# Pass through
qm set 100 --hostpci3 03:00.0,pcie=1

Network Card

Pass dedicated NIC:

Terminal window
qm set 100 --hostpci4 04:00.0,pcie=1

Troubleshooting

Device Not Bound to VFIO

Terminal window
# Check current driver
lspci -nnk -s 01:00


# If not vfio-pci:
# 1. Check blacklist
cat /etc/modprobe.d/blacklist.conf


# 2. Check VFIO config
cat /etc/modprobe.d/vfio.conf


# 3. Rebuild initramfs
update-initramfs -u -k all


# 4. Reboot

VM Won’t Start

Terminal window
# Check QEMU log
cat /var/log/pve/qemu-server/100.log


# Common issues:
# - IOMMU not enabled
# - Device in use by host
# - Wrong device ID

No Display Output

Terminal window
# Verify x-vga=1 is set
grep hostpci /etc/pve/qemu-server/100.conf


# Try different video output
# Monitor on GPU should show VM boot


# Check if VM is actually running
qm status 100

NVIDIA Error 43

Terminal window
# Verify hidden flag
grep cpu /etc/pve/qemu-server/100.conf
# Should include hidden=1


# Try vendor ID spoof
# Add to args: hv_vendor_id=NV43FIX


# Ensure BIOS is OVMF, not SeaBIOS

Poor Performance

Terminal window
# Inside VM:
# Check if GPU is using correct driver
nvidia-smi  # Should show GPU


# Check PCIe link speed
lspci -vv -s 01:00 | grep -i width
# Should show x16 or at least x8


# Ensure cpu type is 'host'

Single GPU Passthrough

Using your only GPU in a VM (no display on host):

Terminal window
# Host boots headless
# VM gets GPU
# Reconnect display to VM


# Challenges:
# - Host has no display
# - Must manage via SSH/remote
# - GPU must unbind from host console


# Scripts needed for:
# 1. Unbind GPU from host
# 2. Start VM
# 3. Stop VM
# 4. Rebind GPU to host

This is complex. Search for “single GPU passthrough scripts” for examples.

Live Migration Limitations

VMs with passthrough cannot live migrate:

  • Hardware is physically on one host
  • Must stop VM, move, start on new host
  • Not compatible with HA auto-failover

Plan accordingly: critical passthrough VMs can’t be HA.

The Lesson

Passthrough is hardware compatibility plus attention to detail.

When passthrough doesn’t work, it’s usually:

  1. IOMMU not enabled: Check BIOS and kernel parameters
  2. Bad IOMMU groups: ACS override or different hardware
  3. Driver conflict: Host driver grabs device before VFIO
  4. NVIDIA detection: Hidden flags and vendor ID spoof
  5. Reset bugs: AMD GPUs need vendor-reset module

The debugging process:

  1. Verify IOMMU enabled (dmesg | grep IOMMU)
  2. Check IOMMU groups (all needed devices in one group)
  3. Verify VFIO binding (lspci -nnk)
  4. Check VM logs (/var/log/pve/qemu-server/)
  5. Try minimal config, add features one at a time

Passthrough isn’t guaranteed to work with all hardware. Some motherboards have terrible IOMMU groups. Some GPUs have bugs. Do research before buying hardware specifically for passthrough.

When it works, you get bare-metal GPU performance in a VM. When it doesn’t, you need patience and systematic debugging. There’s no magic fix — just working through each requirement methodically.