From fbf5ce2616108574a0f963a2dd1b32f2d64465dd Mon Sep 17 00:00:00 2001 From: Miguel Palhas Date: Mon, 17 Aug 2026 17:48:41 +0100 Subject: [PATCH] fix: stream real frames from yolo instead of a black picture Three separate faults, each producing a black Moonlight stream: - sunshine's wlr backend takes wlr-screencopy's dmabuf path, which on this virtio-gpu returns empty buffers. Reproduced locally over 127.0.0.1, so it is not the network or the client. Switch to kms capture. - hypridle locked the session and cut dpms, blanking the captured output with no console to unlock from. Disable it on this host. - qemu picked the nvidia render node by readdir order, so virgl fell back to llvmpipe. Pin renderD128 via the VM's args. Co-Authored-By: Claude Opus 5 (1M context) --- home/yolo/default.nix | 4 ++ hosts/common/features/remote-desktop.nix | 12 +++-- hosts/yolo/PROXMOX.md | 68 +++++++++++------------- 3 files changed, 45 insertions(+), 39 deletions(-) diff --git a/home/yolo/default.nix b/home/yolo/default.nix index ea9a9c9..629012b 100644 --- a/home/yolo/default.nix +++ b/home/yolo/default.nix @@ -39,6 +39,10 @@ programs.agentSkills.machine = "yolo"; programs.nh.flake = lib.mkForce "/home/naps62/tea/nixos-config"; + # Idle lock and dpms-off blank the virtual output: Sunshine then captures a + # flat frame and Moonlight goes black, with no console to unlock from. + services.hypridle.enable = lib.mkForce false; + # Blur and shadow cost a fullscreen pass per frame, and every frame here is # also x264-encoded for the stream — on a virtio-gpu with no VirGL, in software. wayland.windowManager.hyprland.extraConfig = '' diff --git a/hosts/common/features/remote-desktop.nix b/hosts/common/features/remote-desktop.nix index ab9abb1..86908de 100644 --- a/hosts/common/features/remote-desktop.nix +++ b/hosts/common/features/remote-desktop.nix @@ -15,7 +15,7 @@ _: enable = true; autoStart = true; openFirewall = true; - capSysAdmin = false; # wlr-screencopy needs no CAP_SYS_ADMIN + capSysAdmin = true; # kms capture reads the scanout framebuffer via DRM settings = { # MUST be set. Auto-probe tries portalgrab first, and @@ -23,11 +23,17 @@ _: # probe then hangs forever instead of falling back, so sunshine never # binds its ports and the unit sits "active" doing nothing. # - # "wlr", not "wlgrab": wlgrab is the name this backend logs under, but it + # "kms", not "wlr": the wlr backend takes wlr-screencopy's dmabuf path, + # which on this virtio-gpu returns empty buffers — sunshine encodes a flat + # frame (~50-byte P-frames) and every client shows black, with no error on + # either side. grim still works because it uses the shm path, and sunshine + # exposes no way to force shm. + # + # "wlr", not "wlgrab": wlgrab is the name that backend logs under, but it # is not accepted as a value — sunshine rejects it with "Unable to # initialize capture method", then still starts and binds its ports with # no encoder, so the failure only shows up as a 503 in moonlight. - capture = "wlr"; + capture = "kms"; # The web UI is only reachable over the network here — there is no local # browser — and sunshine CSRF-rejects any origin but localhost unless it diff --git a/hosts/yolo/PROXMOX.md b/hosts/yolo/PROXMOX.md index f5f1224..e9978d7 100644 --- a/hosts/yolo/PROXMOX.md +++ b/hosts/yolo/PROXMOX.md @@ -17,7 +17,8 @@ qm create 132 --name yolo \ --vga virtio-gl \ --serial0 socket \ --agent 1 \ - --onboot 1 + --onboot 1 \ + --args '-display egl-headless,gl=core,rendernode=/dev/dri/renderD128' ``` After creating the disk, drop its ZFS reservation — the `data` pool has no @@ -27,12 +28,15 @@ After creating the disk, drop its ZFS reservation — the `data` pool has no zfs set refreservation=none data/vm-132-disk-1 ``` -## The three that matter +## The four that matter -**`--vga virtio-gl`.** Not for speed — the host's virglrenderer falls back to -llvmpipe either way. It is what lets mesa allocate GBM buffers. With plain -`virtio` every capture fails with `Failed to create GBM buffer`, and sunshine -streams a black picture while reporting no error at all. +**`--vga virtio-gl`.** Lets mesa allocate GBM buffers. With plain `virtio` +every capture fails with `Failed to create GBM buffer`, and sunshine streams a +black picture while reporting no error at all. + +**`--args '-display egl-headless,...,rendernode=/dev/dri/renderD128'`.** Pins +host-side rendering to the UHD 770. Without it the guest renders on llvmpipe — +see below. **`--serial0 socket`.** `virtio-gl` renders through a GL context with no QEMU console surface, so the Proxmox noVNC console and `qm screendump` both go dark. @@ -42,44 +46,36 @@ This plus `boot.kernelParams = [ "console=ttyS0,115200" ]` keeps **`--bios ovmf`.** `hosts/common/features/user.nix` uses systemd-boot, which needs UEFI. SeaBIOS gives an unbootable disk. -## Known limit: no GPU acceleration +## Why the render node has to be pinned -The guest reports `virgl (LLVMPIPE)`, so compositing and capture readback are -both on CPU. Hyprland delivers ~2fps at 2560x1440, sunshine asks for 60, and -moonlight disconnects. Capture and encoding are otherwise fine — x264 manages -1440p at 3x realtime here. +pve 8.4.1 hardcodes `-display egl-headless,gl=core` in +`/usr/share/perl5/PVE/QemuServer.pm` with no `rendernode=`. qemu then picks a +node itself by scanning `/dev/dri` in `readdir` order and taking the first +`renderD*` that opens. Here that lands on renderD129 — the nvidia card, which +is reserved for LXCs. mesa cannot drive it, so virglrenderer falls back to +software and the guest reports `virgl (LLVMPIPE)`: Hyprland manages ~2fps at +2560x1440 while sunshine asks for 60, and moonlight disconnects. -The host can do better: `iris_dri.so` is present and `i915` is loaded on -renderD128 (the UHD 770; renderD129 is the nvidia card, reserved for LXCs and -not available to VMs). Two things stop it being used: - -- pve 8.4.1 hardcodes `-display egl-headless,gl=core` in - `/usr/share/perl5/PVE/QemuServer.pm` with no `rendernode=`, so qemu picks the - EGL device itself. -- `/usr/share/glvnd/egl_vendor.d/` lists `10_nvidia.json` ahead of - `50_mesa.json`, and lower wins — so EGL initialises against nvidia and falls - back to llvmpipe. - -To test the diagnosis without changing anything, start the VM with the mesa -vendor forced: +The `--args` line overrides this. PVE appends `args` after its own `-display`, +and qemu's last `-display` wins, so rendering is pinned to renderD128 (the UHD +770, `i915`) without touching the packaged perl — which any pve-manager upgrade +would revert. Confirm from the guest: ``` -__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json qm start 132 +grep "Renderer:" /run/user/1000/hypr/*/hyprland.log ``` -If the guest then reports `iris` rather than `LLVMPIPE`, persist it with a -drop-in rather than by editing the packaged perl — drop-ins survive upgrades, -and the nvidia LXCs are unaffected because they start via pve-container: +`virgl (Mesa Intel(R) Graphics (RPL-S))` is correct; `LLVMPIPE` means the +override is not taking. On the host, `ls -l /proc/$(cat +/var/run/qemu-server/132.pid)/fd | grep dri` shows which node qemu actually +holds. -``` -systemctl edit pvedaemon - [Service] - Environment=__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json -``` +Forcing the EGL vendor instead (`__EGL_VENDOR_LIBRARY_FILENAMES=.../50_mesa.json`) +does not help and was tried: the vendor was never the problem, the node was. -Last resort only, if EGL still picks wrong: append -`,rendernode=/dev/dri/renderD128` to the QemuServer.pm line above. That edits a -packaged file and is reverted by any pve-manager upgrade. +Encoding stays on x264. virgl accelerates GL only — the guest sees a +virtio-gpu, not the Intel device, so there is no VAAPI/QuickSync encode path. +Not a bottleneck: x264 manages 1440p at 3x realtime here. ## Network