15 hours of downtime: the graphics card did it

OVH sent me an email. My VPS was getting a faster line, 400 Mbit up to 1 Gbit, and all I had to do was reboot it. Nice.

I measured first, because I wanted to see the difference afterwards. 41 MB/s. That is 400 Mbit, so the old number checked out at least.

So I typed sudo reboot in a good mood and settled in for the usual fifteen seconds.

And SSH did not come back. I gave it two or three minutes before I admitted something was wrong.

But the OVH panel said Active. As far as they were concerned the machine was fine. So I pressed Reboot there, the obvious thing to do, and that is where it properly died. Fifteen hours.

I tried the KVM console and SSH and got nothing out of either. But pings came back, and that was the only sign of life the whole time. In hindsight that was the kernel telling me it was still there and everything above it was not. I opened a ticket and then just read about what causes this kind of thing, because OVH support does not have the best reputation and I was expecting a week of it. That evening I started looking at other hosts.

I did not do much that night. I read. Does this happen to guests at all, could it be something on the hypervisor side, what could I possibly have misconfigured on a clean Debian install. That last one kept coming back and I never had a good answer to it, which is roughly how I talked myself into blaming the host.

And it came back at eight the next morning, restarted by hand at their end. I checked the line first. 1 Gbit. Good.

So, the short version, because I took far too long to get there myself. It was not the hypervisor. It is a known bug in the guest's graphics driver, and the kernel has known about it for years and ships it anyway, on purpose.

The machine is a VPS at OVH in Zurich, Debian, every service in Docker containers, no Kubernetes. None of that is exotic. That is why this hits a whole class of servers. The exact numbers come later, where the blame is handed out.

If you have a wedged machine in front of you right now and are not here for my story, the way out without a hard reset is under Out without systemd and the permanent fix under What I changed. The rest keeps until the box is back.

The error I was not looking for#

Then I remembered OVH have a 2027 range with slightly worse specs, slower memory and a weaker CPU, and I got nervous they had quietly moved me onto it. So I ran benchmarks. The benchmarks were fine.

But while I was installing iperf3, this came up in red.

Reload daemon failed: Transport endpoint is not connected

I had no idea what that had to do with anything. It is a normal package. systemctl is-system-running gave me the same thing, and it hung rather than failing. No timeout, no error. I googled it. Dead systemd.

And that is when it hit me. One more ordinary reboot and the machine goes straight back into that state, and the hypervisor cannot get it out. So check PID 1:

ps -p 1 -o pid,comm,stat

   PID COMMAND STAT
      1 systemd Ds

Ds. Uninterruptible sleep. The process sits inside a kernel call and nothing gets it out, not even SIGKILL, because signals are not delivered in that state at all. I knew that in theory. Meeting it on PID 1 is different.

Everything looked fine from outside, by the way. Containers running, services answering, any uptime check green. The broken part was the one nobody checks from outside.

So I went after the boring explanation first, the one I wanted it to be. A descriptor leak, some limit hit.

ls /proc/1/fd | wc -l
# 127
cat /proc/1/limits | grep -i "open files"
# Max open files            1073741816           1073741816           files

127 open, limit in the millions. Not that either. So, dmesg:

dmesg -T | tail -40

The same line about twenty times over.

[TTM] Buffer eviction failed

TTM is the memory management layer under DRM. That is the graphics pipeline, on a server with no screen.

Where PID 1 was stuck#

cat /proc/1/wchan; echo
cat /proc/1/syscall
cat /proc/1/stack 2>/dev/null | head -20

wchan was drm_modeset_lock. The syscall was a write. The stack, from the caller down to the innermost frame:

write()                            systemd writing to /dev/console
  n_tty_write, con_write           TTY
  do_con_write, fbcon_switch       framebuffer console
  bit_update_start, fb_pan_display
  drm_fb_helper_pan_display        DRM
  drm_client_modeset_commit_locked
  drm_client_modeset_commit_atomic
  drm_atomic_get_plane_state
  drm_modeset_lock                 everything stops here

systemd tried to write something to the console. The ordinary write() to /dev/console went through the framebuffer console, the framebuffer console talked to DRM, and there sat the mutex that was never released. The same subsystem that was filling dmesg with Buffer eviction failed.

So that was that. The graphics driver was wedged and the whole console path with it. Any process that writes to the console also ends up in D state, and PID 1 was simply the first of them. Filtering the process list shows who else got caught:

ps -eo pid,stat,comm | awk '$2 ~ /^D/'

Out without systemd#

systemctl reboot and shutdown go through the same systemd that was not answering. They would walk into the same deadlock as the reboot the day before, and this time I knew what that costs. Fifteen hours and a support ticket. A hard reset from the panel was out too, as long as data was sitting in the write cache.

Docker runs independently of systemd. The containers could be stopped cleanly:

docker ps
docker stop $(docker ps -q)
sync; sync

One warning before the sequence. This is an emergency route. If systemctl reboot still answers on your machine, take that instead. sysrq talks straight to the kernel and skips every shutdown hook, every unit and every script that would otherwise run on the way down. That is why it works here. It is also why you do not want it anywhere else.

Then the Magic SysRq sequence, which the kernel executes itself, without systemd. The order is described in the kernel documentation:

echo 1 > /proc/sys/kernel/sysrq
echo s > /proc/sysrq-trigger   # sync
sleep 5
echo u > /proc/sysrq-trigger   # remount read-only
sleep 5
echo b > /proc/sysrq-trigger   # reboot

The 1 unlocks every sysrq function, not just the three used here. If you do not want that left open permanently, set the value back to what it was after the restart.

Pressing b was frightening. The backups were done by then and I was already half packed for another host, so I decided to just find out.

SSH drops immediately on b. That is expected, the filesystems are synced and read-only by then. It is far safer than the hard reset from the panel.

Open the OVH panel beforehand, but without much hope. I pressed Reboot there about twenty times. Then I went into their API console looking for the task, and I could see it sitting there, scheduled, state pending, with nothing anywhere to cancel it. The two endpoints I actually wanted were right next to it, /vps/{serviceName}/stop and /vps/{serviceName}/start, and both refused, because another task was already running. That is the part that got to me. I could not hard stop my own machine and start it again, because the request to reboot it had wedged and nothing would clear it. In the end it took a human at the provider. If nothing moves after ten minutes, open a ticket instead of pressing the button again.

After the reboot#

uptime
systemctl is-system-running
# running
systemctl --failed
# 0 loaded units listed.
docker ps

systemd was responding again, and the containers were running. One was missing and had to be started by hand. In dmesg it now said:

[drm] Initialized qxl 0.1.0 for 0000:00:01.0 on minor 0
fbcon: qxldrmfb (fb0) is primary device
qxl 0000:00:01.0: [drm] fb0: qxldrmfb frame buffer device

That driver is qxl. That is QEMU's emulated graphics card for SPICE, and nobody has ever opened a SPICE session to this machine.

Whether the same card sits in your VM is two lines away:

lspci | grep -i vga
lsmod | grep qxl

What I changed#

On a headless VPS you need neither qxl nor the rest of the DRM stack. There are two ways to do it, and a third that gets most readers there faster.

The easy way#

If your provider lets you change the graphics model in the panel, switch it from qxl to virtio-gpu and stop reading here. That is also the advice that keeps coming up in the bug reports further down. At OVH I did not have that option, so the guest was the only way left.

Serial console#

Before I touch the graphics stack, I want a second way onto the machine. The existing line in /etc/default/grub was:

GRUB_CMDLINE_LINUX_DEFAULT="hugepages=1024 "

hugepages=1024 reserves 2 GB for huge pages. I put that in by hand for Postgres, and it stays. The serial console goes on top:

cp /etc/default/grub /etc/default/grub.bak
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="hugepages=1024 "/GRUB_CMDLINE_LINUX_DEFAULT="hugepages=1024 console=tty0 console=ttyS0,115200"/' /etc/default/grub

console=tty0 console=ttyS0,115200 sends kernel output to both consoles. The one named last gets /dev/console, so in an emergency the messages land on the serial line.

Locking the module out#

For the driver itself there are two variants. nomodeset as a kernel parameter stops the kernel from taking over modesetting at all, which hits every DRM driver. Locking out the single module is more precise:

echo "blacklist qxl" > /etc/modprobe.d/blacklist-qxl.conf
echo "# qxl DRM stack deadlocked systemd on 2026-08-03, see drm_modeset_lock" >> /etc/modprobe.d/blacklist-qxl.conf

update-initramfs -u
update-grub

blacklist prevents the automatic load through the PCI identifier, but not a load as a dependency or an explicit modprobe qxl. If you want to be completely sure, use install qxl /bin/true instead, which kills any attempt to load it. For this case the blacklist is enough, because qxl only arrives here over the PCI bus.

Check before the reboot:

grep -c "console=ttyS0" /boot/grub/grub.cfg
# > 0

Then a normal reboot. This time it works, because systemd is healthy again.

After boot:

lsmod | grep qxl
# (empty)
dmesg -T | grep -ci drm
# 2

Two lines are left instead of hundreds. I still get text on the KVM console in the panel, now through vgacon instead of the framebuffer. The serial path is there as a reserve.

The specifics#

Before the blame, the exact numbers:

That kernel version is not a detail, it is the whole story. The commit behind the bug was reverted, put back, and turned around again in individual distributions. Which variant sits on your machine is decided by uname -r.

The blame#

I suspected the hypervisor first, and I was fairly sure about it. The reasoning sounded good. A guest cannot break its own emulated device from the inside, so the host must have broken it. It is wrong, and you can look that up.

The driver in the guest#

The bug is old and documented. In Debian bug #1054514 a reporter narrowed it down in 2023, with a reproducer and a bisect session, to commit 5a838e5d5825, "drm/qxl: simplify qxl_fence_wait", in the tree since 5.13. The symptoms there are exactly mine: [TTM] Buffer eviction failed every few seconds and processes stuck inside the kernel.

What happened next is the interesting part. The commit was reverted with 07ed11afb68d, and that revert brought a different deadlock between the console lock and the worker pool, on which test machines stopped booting entirely. So Linus undid the revert again with 3628e0383dd3 in 6.9, with a remarkably candid justification: this may bring the Buffer eviction failed messages back, and that beats a system that does not boot. Hard to argue with.

The Debian Security Tracker entry for CVE-2024-36944 carries the versions each branch sits at. That table lists 6.12.100-1 for trixie security. That is what runs on this machine. So I did not hit rare bad luck, I got the documented state.

That this is not settled shows in Debian bug #1139265, filed against 7.0.10+deb14-amd64 and therefore on a kernel a good deal newer than mine. Somebody else describes the same chain through fbcon_switch and fb_pan_display down to drm_modeset_lock, the same processes in D state and the same message repeating.

What surprised me was that this is still live. Not a bug from 2023 that got fixed, a bug from 2023 that upstream looked at, weighed, and decided to keep. Which also means I am probably not the only one sitting on it.

OVH's share#

Not the deadlock, but the device. Giving a headless VPS a qxl card is a questionable default. qxl comes from desktop virtualisation with SPICE. On a server without a monitor, virtio-gpu, plain VGA or nothing at all is enough, and the console runs over serial. You trade stability for hardware you never use, and the panel offers no way to swap it. Then there is the reboot button in that same panel, which does not restart a hung machine, and an API that will not let you stop the thing yourself while a wedged task sits in the queue holding the door. The guest caused the hang. That queue is what stretched it to fifteen hours.

That night I was sure OVH were simply an unstable provider. They are not, and the hang was never theirs. What they have is an unfinished panel and too little control over your own VPS. With a stop and a start that actually worked I would probably never have opened a ticket at all, and this would have been twenty minutes instead of fifteen hours.

The acquittals#

Debian loads DRM drivers for everything that appears on the PCI bus. That is a reasonable default for desktop machines and not the distributor's fault. And hugepages=1024 is mine, I wanted those pages for Postgres, but they have nothing to do with this stack. It hangs exclusively on graphics and TTY.

Row two is the part I actually resent. A driver that knows its device is dead should hand the caller an error and let go of the lock. This one keeps it. Every console write behind it then hangs forever, and the first process in that queue is PID 1.

What remains#

So the fix sits in GRUB and in one file under /etc/modprobe.d. That is what kernel parameters and modprobe configuration are for. You take the hardware you do not need, and that can break, out of the game before it takes PID 1 with it.

The lesson I keep is less comfortable. I had a suspect, a plausible explanation and no proof, and I came close to publishing it that way. Between "it must have been the host" and a bisect result with a commit ID lie ten minutes of searching.

If you see the same stack, write to me. Especially if your kernel is newer than mine and the chain still hangs, I would like to know that before it happens to me a second time.