fix: stream real frames from yolo instead of a black picture

Three separate faults, each producing a black Moonlight stream:

- sunshine's wlr backend takes wlr-screencopy's dmabuf path, which on this
  virtio-gpu returns empty buffers. Reproduced locally over 127.0.0.1, so it
  is not the network or the client. Switch to kms capture.
- hypridle locked the session and cut dpms, blanking the captured output with
  no console to unlock from. Disable it on this host.
- qemu picked the nvidia render node by readdir order, so virgl fell back to
  llvmpipe. Pin renderD128 via the VM's args.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Miguel Palhas
2026-08-17 17:48:41 +01:00
parent df319322b9
commit fbf5ce2616
3 changed files with 45 additions and 39 deletions
+4
View File
@@ -39,6 +39,10 @@
programs.agentSkills.machine = "yolo"; programs.agentSkills.machine = "yolo";
programs.nh.flake = lib.mkForce "/home/naps62/tea/nixos-config"; programs.nh.flake = lib.mkForce "/home/naps62/tea/nixos-config";
# Idle lock and dpms-off blank the virtual output: Sunshine then captures a
# flat frame and Moonlight goes black, with no console to unlock from.
services.hypridle.enable = lib.mkForce false;
# Blur and shadow cost a fullscreen pass per frame, and every frame here is # Blur and shadow cost a fullscreen pass per frame, and every frame here is
# also x264-encoded for the stream — on a virtio-gpu with no VirGL, in software. # also x264-encoded for the stream — on a virtio-gpu with no VirGL, in software.
wayland.windowManager.hyprland.extraConfig = '' wayland.windowManager.hyprland.extraConfig = ''
+9 -3
View File
@@ -15,7 +15,7 @@ _:
enable = true; enable = true;
autoStart = true; autoStart = true;
openFirewall = true; openFirewall = true;
capSysAdmin = false; # wlr-screencopy needs no CAP_SYS_ADMIN capSysAdmin = true; # kms capture reads the scanout framebuffer via DRM
settings = { settings = {
# MUST be set. Auto-probe tries portalgrab first, and # MUST be set. Auto-probe tries portalgrab first, and
@@ -23,11 +23,17 @@ _:
# probe then hangs forever instead of falling back, so sunshine never # probe then hangs forever instead of falling back, so sunshine never
# binds its ports and the unit sits "active" doing nothing. # binds its ports and the unit sits "active" doing nothing.
# #
# "wlr", not "wlgrab": wlgrab is the name this backend logs under, but it # "kms", not "wlr": the wlr backend takes wlr-screencopy's dmabuf path,
# which on this virtio-gpu returns empty buffers — sunshine encodes a flat
# frame (~50-byte P-frames) and every client shows black, with no error on
# either side. grim still works because it uses the shm path, and sunshine
# exposes no way to force shm.
#
# "wlr", not "wlgrab": wlgrab is the name that backend logs under, but it
# is not accepted as a value — sunshine rejects it with "Unable to # is not accepted as a value — sunshine rejects it with "Unable to
# initialize capture method", then still starts and binds its ports with # initialize capture method", then still starts and binds its ports with
# no encoder, so the failure only shows up as a 503 in moonlight. # no encoder, so the failure only shows up as a 503 in moonlight.
capture = "wlr"; capture = "kms";
# The web UI is only reachable over the network here — there is no local # The web UI is only reachable over the network here — there is no local
# browser — and sunshine CSRF-rejects any origin but localhost unless it # browser — and sunshine CSRF-rejects any origin but localhost unless it
+32 -36
View File
@@ -17,7 +17,8 @@ qm create 132 --name yolo \
--vga virtio-gl \ --vga virtio-gl \
--serial0 socket \ --serial0 socket \
--agent 1 \ --agent 1 \
--onboot 1 --onboot 1 \
--args '-display egl-headless,gl=core,rendernode=/dev/dri/renderD128'
``` ```
After creating the disk, drop its ZFS reservation — the `data` pool has no After creating the disk, drop its ZFS reservation — the `data` pool has no
@@ -27,12 +28,15 @@ After creating the disk, drop its ZFS reservation — the `data` pool has no
zfs set refreservation=none data/vm-132-disk-1 zfs set refreservation=none data/vm-132-disk-1
``` ```
## The three that matter ## The four that matter
**`--vga virtio-gl`.** Not for speed — the host's virglrenderer falls back to **`--vga virtio-gl`.** Lets mesa allocate GBM buffers. With plain `virtio`
llvmpipe either way. It is what lets mesa allocate GBM buffers. With plain every capture fails with `Failed to create GBM buffer`, and sunshine streams a
`virtio` every capture fails with `Failed to create GBM buffer`, and sunshine black picture while reporting no error at all.
streams a black picture while reporting no error at all.
**`--args '-display egl-headless,...,rendernode=/dev/dri/renderD128'`.** Pins
host-side rendering to the UHD 770. Without it the guest renders on llvmpipe —
see below.
**`--serial0 socket`.** `virtio-gl` renders through a GL context with no QEMU **`--serial0 socket`.** `virtio-gl` renders through a GL context with no QEMU
console surface, so the Proxmox noVNC console and `qm screendump` both go dark. console surface, so the Proxmox noVNC console and `qm screendump` both go dark.
@@ -42,44 +46,36 @@ This plus `boot.kernelParams = [ "console=ttyS0,115200" ]` keeps
**`--bios ovmf`.** `hosts/common/features/user.nix` uses systemd-boot, which **`--bios ovmf`.** `hosts/common/features/user.nix` uses systemd-boot, which
needs UEFI. SeaBIOS gives an unbootable disk. needs UEFI. SeaBIOS gives an unbootable disk.
## Known limit: no GPU acceleration ## Why the render node has to be pinned
The guest reports `virgl (LLVMPIPE)`, so compositing and capture readback are pve 8.4.1 hardcodes `-display egl-headless,gl=core` in
both on CPU. Hyprland delivers ~2fps at 2560x1440, sunshine asks for 60, and `/usr/share/perl5/PVE/QemuServer.pm` with no `rendernode=`. qemu then picks a
moonlight disconnects. Capture and encoding are otherwise fine — x264 manages node itself by scanning `/dev/dri` in `readdir` order and taking the first
1440p at 3x realtime here. `renderD*` that opens. Here that lands on renderD129 — the nvidia card, which
is reserved for LXCs. mesa cannot drive it, so virglrenderer falls back to
software and the guest reports `virgl (LLVMPIPE)`: Hyprland manages ~2fps at
2560x1440 while sunshine asks for 60, and moonlight disconnects.
The host can do better: `iris_dri.so` is present and `i915` is loaded on The `--args` line overrides this. PVE appends `args` after its own `-display`,
renderD128 (the UHD 770; renderD129 is the nvidia card, reserved for LXCs and and qemu's last `-display` wins, so rendering is pinned to renderD128 (the UHD
not available to VMs). Two things stop it being used: 770, `i915`) without touching the packaged perl — which any pve-manager upgrade
would revert. Confirm from the guest:
- pve 8.4.1 hardcodes `-display egl-headless,gl=core` in
`/usr/share/perl5/PVE/QemuServer.pm` with no `rendernode=`, so qemu picks the
EGL device itself.
- `/usr/share/glvnd/egl_vendor.d/` lists `10_nvidia.json` ahead of
`50_mesa.json`, and lower wins — so EGL initialises against nvidia and falls
back to llvmpipe.
To test the diagnosis without changing anything, start the VM with the mesa
vendor forced:
``` ```
__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json qm start 132 grep "Renderer:" /run/user/1000/hypr/*/hyprland.log
``` ```
If the guest then reports `iris` rather than `LLVMPIPE`, persist it with a `virgl (Mesa Intel(R) Graphics (RPL-S))` is correct; `LLVMPIPE` means the
drop-in rather than by editing the packaged perl — drop-ins survive upgrades, override is not taking. On the host, `ls -l /proc/$(cat
and the nvidia LXCs are unaffected because they start via pve-container: /var/run/qemu-server/132.pid)/fd | grep dri` shows which node qemu actually
holds.
``` Forcing the EGL vendor instead (`__EGL_VENDOR_LIBRARY_FILENAMES=.../50_mesa.json`)
systemctl edit pvedaemon does not help and was tried: the vendor was never the problem, the node was.
[Service]
Environment=__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json
```
Last resort only, if EGL still picks wrong: append Encoding stays on x264. virgl accelerates GL only — the guest sees a
`,rendernode=/dev/dri/renderD128` to the QemuServer.pm line above. That edits a virtio-gpu, not the Intel device, so there is no VAAPI/QuickSync encode path.
packaged file and is reverted by any pve-manager upgrade. Not a bottleneck: x264 manages 1440p at 3x realtime here.
## Network ## Network