fix: stream real frames from yolo instead of a black picture
Three separate faults, each producing a black Moonlight stream: - sunshine's wlr backend takes wlr-screencopy's dmabuf path, which on this virtio-gpu returns empty buffers. Reproduced locally over 127.0.0.1, so it is not the network or the client. Switch to kms capture. - hypridle locked the session and cut dpms, blanking the captured output with no console to unlock from. Disable it on this host. - qemu picked the nvidia render node by readdir order, so virgl fell back to llvmpipe. Pin renderD128 via the VM's args. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -39,6 +39,10 @@
|
|||||||
programs.agentSkills.machine = "yolo";
|
programs.agentSkills.machine = "yolo";
|
||||||
programs.nh.flake = lib.mkForce "/home/naps62/tea/nixos-config";
|
programs.nh.flake = lib.mkForce "/home/naps62/tea/nixos-config";
|
||||||
|
|
||||||
|
# Idle lock and dpms-off blank the virtual output: Sunshine then captures a
|
||||||
|
# flat frame and Moonlight goes black, with no console to unlock from.
|
||||||
|
services.hypridle.enable = lib.mkForce false;
|
||||||
|
|
||||||
# Blur and shadow cost a fullscreen pass per frame, and every frame here is
|
# Blur and shadow cost a fullscreen pass per frame, and every frame here is
|
||||||
# also x264-encoded for the stream — on a virtio-gpu with no VirGL, in software.
|
# also x264-encoded for the stream — on a virtio-gpu with no VirGL, in software.
|
||||||
wayland.windowManager.hyprland.extraConfig = ''
|
wayland.windowManager.hyprland.extraConfig = ''
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ _:
|
|||||||
enable = true;
|
enable = true;
|
||||||
autoStart = true;
|
autoStart = true;
|
||||||
openFirewall = true;
|
openFirewall = true;
|
||||||
capSysAdmin = false; # wlr-screencopy needs no CAP_SYS_ADMIN
|
capSysAdmin = true; # kms capture reads the scanout framebuffer via DRM
|
||||||
|
|
||||||
settings = {
|
settings = {
|
||||||
# MUST be set. Auto-probe tries portalgrab first, and
|
# MUST be set. Auto-probe tries portalgrab first, and
|
||||||
@@ -23,11 +23,17 @@ _:
|
|||||||
# probe then hangs forever instead of falling back, so sunshine never
|
# probe then hangs forever instead of falling back, so sunshine never
|
||||||
# binds its ports and the unit sits "active" doing nothing.
|
# binds its ports and the unit sits "active" doing nothing.
|
||||||
#
|
#
|
||||||
# "wlr", not "wlgrab": wlgrab is the name this backend logs under, but it
|
# "kms", not "wlr": the wlr backend takes wlr-screencopy's dmabuf path,
|
||||||
|
# which on this virtio-gpu returns empty buffers — sunshine encodes a flat
|
||||||
|
# frame (~50-byte P-frames) and every client shows black, with no error on
|
||||||
|
# either side. grim still works because it uses the shm path, and sunshine
|
||||||
|
# exposes no way to force shm.
|
||||||
|
#
|
||||||
|
# "wlr", not "wlgrab": wlgrab is the name that backend logs under, but it
|
||||||
# is not accepted as a value — sunshine rejects it with "Unable to
|
# is not accepted as a value — sunshine rejects it with "Unable to
|
||||||
# initialize capture method", then still starts and binds its ports with
|
# initialize capture method", then still starts and binds its ports with
|
||||||
# no encoder, so the failure only shows up as a 503 in moonlight.
|
# no encoder, so the failure only shows up as a 503 in moonlight.
|
||||||
capture = "wlr";
|
capture = "kms";
|
||||||
|
|
||||||
# The web UI is only reachable over the network here — there is no local
|
# The web UI is only reachable over the network here — there is no local
|
||||||
# browser — and sunshine CSRF-rejects any origin but localhost unless it
|
# browser — and sunshine CSRF-rejects any origin but localhost unless it
|
||||||
|
|||||||
+32
-36
@@ -17,7 +17,8 @@ qm create 132 --name yolo \
|
|||||||
--vga virtio-gl \
|
--vga virtio-gl \
|
||||||
--serial0 socket \
|
--serial0 socket \
|
||||||
--agent 1 \
|
--agent 1 \
|
||||||
--onboot 1
|
--onboot 1 \
|
||||||
|
--args '-display egl-headless,gl=core,rendernode=/dev/dri/renderD128'
|
||||||
```
|
```
|
||||||
|
|
||||||
After creating the disk, drop its ZFS reservation — the `data` pool has no
|
After creating the disk, drop its ZFS reservation — the `data` pool has no
|
||||||
@@ -27,12 +28,15 @@ After creating the disk, drop its ZFS reservation — the `data` pool has no
|
|||||||
zfs set refreservation=none data/vm-132-disk-1
|
zfs set refreservation=none data/vm-132-disk-1
|
||||||
```
|
```
|
||||||
|
|
||||||
## The three that matter
|
## The four that matter
|
||||||
|
|
||||||
**`--vga virtio-gl`.** Not for speed — the host's virglrenderer falls back to
|
**`--vga virtio-gl`.** Lets mesa allocate GBM buffers. With plain `virtio`
|
||||||
llvmpipe either way. It is what lets mesa allocate GBM buffers. With plain
|
every capture fails with `Failed to create GBM buffer`, and sunshine streams a
|
||||||
`virtio` every capture fails with `Failed to create GBM buffer`, and sunshine
|
black picture while reporting no error at all.
|
||||||
streams a black picture while reporting no error at all.
|
|
||||||
|
**`--args '-display egl-headless,...,rendernode=/dev/dri/renderD128'`.** Pins
|
||||||
|
host-side rendering to the UHD 770. Without it the guest renders on llvmpipe —
|
||||||
|
see below.
|
||||||
|
|
||||||
**`--serial0 socket`.** `virtio-gl` renders through a GL context with no QEMU
|
**`--serial0 socket`.** `virtio-gl` renders through a GL context with no QEMU
|
||||||
console surface, so the Proxmox noVNC console and `qm screendump` both go dark.
|
console surface, so the Proxmox noVNC console and `qm screendump` both go dark.
|
||||||
@@ -42,44 +46,36 @@ This plus `boot.kernelParams = [ "console=ttyS0,115200" ]` keeps
|
|||||||
**`--bios ovmf`.** `hosts/common/features/user.nix` uses systemd-boot, which
|
**`--bios ovmf`.** `hosts/common/features/user.nix` uses systemd-boot, which
|
||||||
needs UEFI. SeaBIOS gives an unbootable disk.
|
needs UEFI. SeaBIOS gives an unbootable disk.
|
||||||
|
|
||||||
## Known limit: no GPU acceleration
|
## Why the render node has to be pinned
|
||||||
|
|
||||||
The guest reports `virgl (LLVMPIPE)`, so compositing and capture readback are
|
pve 8.4.1 hardcodes `-display egl-headless,gl=core` in
|
||||||
both on CPU. Hyprland delivers ~2fps at 2560x1440, sunshine asks for 60, and
|
`/usr/share/perl5/PVE/QemuServer.pm` with no `rendernode=`. qemu then picks a
|
||||||
moonlight disconnects. Capture and encoding are otherwise fine — x264 manages
|
node itself by scanning `/dev/dri` in `readdir` order and taking the first
|
||||||
1440p at 3x realtime here.
|
`renderD*` that opens. Here that lands on renderD129 — the nvidia card, which
|
||||||
|
is reserved for LXCs. mesa cannot drive it, so virglrenderer falls back to
|
||||||
|
software and the guest reports `virgl (LLVMPIPE)`: Hyprland manages ~2fps at
|
||||||
|
2560x1440 while sunshine asks for 60, and moonlight disconnects.
|
||||||
|
|
||||||
The host can do better: `iris_dri.so` is present and `i915` is loaded on
|
The `--args` line overrides this. PVE appends `args` after its own `-display`,
|
||||||
renderD128 (the UHD 770; renderD129 is the nvidia card, reserved for LXCs and
|
and qemu's last `-display` wins, so rendering is pinned to renderD128 (the UHD
|
||||||
not available to VMs). Two things stop it being used:
|
770, `i915`) without touching the packaged perl — which any pve-manager upgrade
|
||||||
|
would revert. Confirm from the guest:
|
||||||
- pve 8.4.1 hardcodes `-display egl-headless,gl=core` in
|
|
||||||
`/usr/share/perl5/PVE/QemuServer.pm` with no `rendernode=`, so qemu picks the
|
|
||||||
EGL device itself.
|
|
||||||
- `/usr/share/glvnd/egl_vendor.d/` lists `10_nvidia.json` ahead of
|
|
||||||
`50_mesa.json`, and lower wins — so EGL initialises against nvidia and falls
|
|
||||||
back to llvmpipe.
|
|
||||||
|
|
||||||
To test the diagnosis without changing anything, start the VM with the mesa
|
|
||||||
vendor forced:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json qm start 132
|
grep "Renderer:" /run/user/1000/hypr/*/hyprland.log
|
||||||
```
|
```
|
||||||
|
|
||||||
If the guest then reports `iris` rather than `LLVMPIPE`, persist it with a
|
`virgl (Mesa Intel(R) Graphics (RPL-S))` is correct; `LLVMPIPE` means the
|
||||||
drop-in rather than by editing the packaged perl — drop-ins survive upgrades,
|
override is not taking. On the host, `ls -l /proc/$(cat
|
||||||
and the nvidia LXCs are unaffected because they start via pve-container:
|
/var/run/qemu-server/132.pid)/fd | grep dri` shows which node qemu actually
|
||||||
|
holds.
|
||||||
|
|
||||||
```
|
Forcing the EGL vendor instead (`__EGL_VENDOR_LIBRARY_FILENAMES=.../50_mesa.json`)
|
||||||
systemctl edit pvedaemon
|
does not help and was tried: the vendor was never the problem, the node was.
|
||||||
[Service]
|
|
||||||
Environment=__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Last resort only, if EGL still picks wrong: append
|
Encoding stays on x264. virgl accelerates GL only — the guest sees a
|
||||||
`,rendernode=/dev/dri/renderD128` to the QemuServer.pm line above. That edits a
|
virtio-gpu, not the Intel device, so there is no VAAPI/QuickSync encode path.
|
||||||
packaged file and is reverted by any pve-manager upgrade.
|
Not a bottleneck: x264 manages 1440p at 3x realtime here.
|
||||||
|
|
||||||
## Network
|
## Network
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user