15 hours of downtime: the graphics card did it

On 3 August I typed reboot over SSH. The machine never came back. Every container gone, every service gone, no SSH. It answered pings and nothing else.

The reboot button in the OVH panel changed none of that. On 4 August at eight in the morning support restarted the machine by hand. Fifteen hours of downtime for a six-letter command.

That is only where the real story starts, because the server was only superficially back. This is the path from the moment packages stopped installing cleanly to the line in /etc/default/grub that prevents it next time. I will give away the ending, because I got to it too late myself: it was not the hypervisor. It is a known bug in the guest's graphics driver, one the kernel has known about for years and today accepts on purpose.

The machine#

A VPS at OVH in Zurich, Debian, every service in Docker containers, no Kubernetes. None of that is exotic, which is exactly why the bug hits a whole class of servers.

The kernel version is not a detail here, it is the core of the story. The commit behind the bug further down was reverted, put back, and turned around again in individual distributions. Which variant sits on your machine is decided by uname -r.

What showed up first#

The server was running again, the containers had come back on their own through restart: unless-stopped, and SSH worked. I wanted to update packages and noticed something was off. systemctl daemon-reload returned:

Reload daemon failed: Transport endpoint is not connected

systemctl is-system-running simply hung. No timeout, no error message. It was stable, not a one-off. So check PID 1:

ps -p 1 -o pid,comm,stat

    PID COMMAND STAT
      1 systemd Ds

Ds means uninterruptible sleep. A process in that state handles no signals, not even SIGKILL. It is stuck inside a kernel syscall and only comes back out when the expected I/O operation finishes, which in this case never happens.

Open file descriptors looked normal:

ls /proc/1/fd | wc -l
# 127
cat /proc/1/limits | grep -i "open files"
# Max open files            1073741816           1073741816           files

No resource exhaustion. Then dmesg:

dmesg -T | tail -40

Every 15 seconds the same line, for minutes:

[TTM] Buffer eviction failed

TTM is the memory management layer under DRM. That is the graphics pipeline, on a server with no screen.

The stack#

cat /proc/1/wchan; echo
cat /proc/1/syscall
cat /proc/1/stack 2>/dev/null | head -20

wchan was drm_modeset_lock. The syscall was a write. The stack, from the caller down to the innermost frame:

write() → tty → n_tty_write → con_write → do_con_write →
fbcon_switch → bit_update_start → fb_pan_display →
drm_fb_helper_pan_display → drm_client_modeset_commit_locked →
drm_client_modeset_commit_atomic → drm_atomic_get_plane_state →
drm_modeset_lock

systemd tried to write something to the console. The ordinary write() to /dev/console went through the framebuffer console, the framebuffer console talked to DRM, and there sat the mutex that was never released. The same subsystem that reports Buffer eviction failed every 15 seconds.

That settles it: the graphics driver is wedged, and with it the entire console path. Any process that writes to the console also ends up in D state, and PID 1 was simply the first of them.

Out without systemd#

systemctl reboot and shutdown go through the same systemd that was not answering. They would walk into the same deadlock as the reboot the day before, and this time I knew the price: fifteen hours and a support ticket. A hard reset from the panel was out too, as long as data was sitting in the write cache.

Docker runs independently of systemd. The containers could be stopped cleanly:

docker ps
docker stop $(docker ps -q)
sync; sync

Then the Magic SysRq sequence, which the kernel executes itself, without systemd. The order is described in the kernel documentation:

echo 1 > /proc/sys/kernel/sysrq
echo s > /proc/sysrq-trigger   # sync
sleep 5
echo u > /proc/sysrq-trigger   # remount read-only
sleep 5
echo b > /proc/sysrq-trigger   # reboot

The 1 unlocks every sysrq function, not just the three used here. If you do not want that left open permanently, set the value back to what it was after the restart.

SSH drops immediately on b. That is expected, the filesystems are synced and read-only by then. It is far safer than the hard reset from the panel.

Open the OVH panel beforehand, but without high hopes. In my case the reboot button there did nothing at all, and what it took in the end was a human at the provider. If you see no reaction after ten minutes, open a ticket rather than keep pressing the same button.

After the reboot#

uptime
systemctl is-system-running
# running
systemctl --failed
# 0 loaded units listed.
docker ps

systemd was responding again, and the containers were running. One was missing and had to be started by hand. In dmesg it now said:

[drm] Initialized qxl 0.1.0 for 0000:00:01.0 on minor 0
fbcon: qxldrmfb (fb0) is primary device
qxl 0000:00:01.0: [drm] fb0: qxldrmfb frame buffer device

The driver being loaded there is qxl. That is QEMU's emulated graphics card for SPICE. No screen is attached to this server, and nobody has ever opened a SPICE session to it.

The fix#

On a headless VPS you need neither qxl nor the rest of the DRM stack. There are two ways to do it, and a third that gets most readers there faster.

The easy way#

If your provider lets you change the graphics model in the panel, switch it from qxl to virtio-gpu and stop reading here. That is also the advice that keeps coming up in the bug reports further down. At OVH I did not have that option, so the guest was the only way left.

Serial console#

Before I touch the graphics stack, I want a second way onto the machine. The existing line in /etc/default/grub was:

GRUB_CMDLINE_LINUX_DEFAULT="hugepages=1024 "

hugepages=1024 reserves 2 GB for huge pages. I put that in by hand for Postgres, and it stays. The serial console goes on top:

cp /etc/default/grub /etc/default/grub.bak
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="hugepages=1024 "/GRUB_CMDLINE_LINUX_DEFAULT="hugepages=1024 console=tty0 console=ttyS0,115200"/' /etc/default/grub

console=tty0 console=ttyS0,115200 sends kernel output to both consoles. The one named last gets /dev/console, so in an emergency the messages land on the serial line.

Locking the module out#

For the driver itself there are two variants. nomodeset as a kernel parameter stops the kernel from taking over modesetting at all, which hits every DRM driver. Locking out the single module is more precise:

echo "blacklist qxl" > /etc/modprobe.d/blacklist-qxl.conf
echo "# qxl DRM stack deadlocked systemd on 2026-08-03, see drm_modeset_lock" >> /etc/modprobe.d/blacklist-qxl.conf

update-initramfs -u
update-grub

blacklist prevents the automatic load through the PCI identifier, but not a load as a dependency or an explicit modprobe qxl. If you want to be completely sure, use install qxl /bin/true instead, which kills any attempt to load it. For this case the blacklist is enough, because qxl only arrives here over the PCI bus.

Check before the reboot:

grep -c "console=ttyS0" /boot/grub/grub.cfg
# > 0

Then a normal reboot. This time it works, because systemd is healthy again.

After boot:

lsmod | grep qxl
# (empty)
dmesg -T | grep -ci drm
# 2

Two lines are left instead of hundreds. The graphical KVM console in the OVH panel still shows text, now through vgacon instead of the framebuffer. The serial path is there as a reserve.

The blame#

I suspected the hypervisor first. The reasoning sounded good: a guest cannot break its own emulated device from the inside, so the host must have broken it. It is wrong, and you can look that up.

The driver in the guest#

The bug is old and documented. In Debian bug #1054514 a reporter narrowed it down in 2023, with a reproducer and a bisect session, to commit 5a838e5d5825, "drm/qxl: simplify qxl_fence_wait", in the tree since 5.13. The symptoms there are exactly mine: [TTM] Buffer eviction failed every few seconds and processes stuck inside the kernel.

What happened next is the interesting part. The commit was reverted with 07ed11afb68d, and that revert brought a different deadlock between the console lock and the worker pool, on which test machines stopped booting entirely. So Linus undid the revert again with 3628e0383dd3 in 6.9, with a remarkably candid justification in the commit message: this may bring the Buffer eviction failed messages back, and that is preferable to a system that does not boot. It is all in the Debian Security Tracker entry for CVE-2024-36944, including the versions each branch sits at. That table lists 6.12.100-1 for trixie security, and that is exactly what runs on this machine. So I did not hit rare bad luck, I got the documented state.

That this is not settled shows in Debian bug #1139265 on a current kernel: somebody else describes the same chain through fbcon_switch and fb_pan_display down to drm_modeset_lock, the same processes in D state and the same message every 15 seconds.

OVH's share#

Not the deadlock, but the device. Giving a headless VPS a qxl card is a questionable default. qxl comes from desktop virtualisation with SPICE. On a server without a monitor, virtio-gpu, plain VGA or nothing at all is enough, and the console runs over serial. You trade stability for a device you never use, and you cannot deselect it in the panel. On top of that, the reboot button in that same panel does not restart a hung machine. The guest caused the hang, and that button is what stretched it out until somebody at the provider stepped in by hand in the morning.

The acquittals#

Debian loads DRM drivers for everything that appears on the PCI bus. That is a reasonable default for desktop machines and not the distributor's fault. And hugepages=1024 is mine, I wanted those pages for Postgres, but they have nothing to do with this stack. It hangs exclusively on graphics and TTY.

The amplifier in row two is the part that bothers me most. A driver that recognises its device as dead should hand the caller back an error. Instead every write to the console hangs here forever, and the first one it catches is PID 1.

What remains#

The fix sits in GRUB and in one file under /etc/modprobe.d. That is exactly what kernel parameters and modprobe configuration are for: taking hardware you do not need, and that can break, out of the game before it takes PID 1 with it.

The real lesson is less comfortable. I had a suspect, a plausible explanation and no proof, and I came close to publishing it that way. Between "it must have been the host" and a bisect result with a commit ID lie ten minutes of searching.

If you see the same stack, write to me. Especially if your kernel is newer than mine and the chain still hangs, I would like to know that before it happens to me a second time.