yolo/arr
3
0
Fork 0
Code Issues 12 Pull Requests Actions Packages Projects Releases Wiki Activity

Decode downloaded subtitles to UTF-8 #212

New Issue
Closed
opened 2026-08-24 22:36:47 +01:00 by naps62-yolo · 1 comment
naps62-yolo commented 2026-08-24 22:36:47 +01:00
Owner
Copy Link
Copy Source

Provider::download returns Fetched.content as Vec<u8> — raw bytes,
deliberately, because providers serve Latin-1 and Windows-1252 files and
guessing wrong at the provider boundary is worse than handing the bytes on
(#184). arr_subs::srt::parse deliberately takes &str (#190).

Nothing in the milestone closes that gap. Every downstream consumer needs
text: translation parses cues, alass needs a file it can read (#194), and
the sidecar written into the title folder has to be in a known encoding
(#195).

Decode once, in arr-subs, between download and everything else:

  • Honour a UTF-8 BOM when present.
  • Otherwise detect the encoding rather than assuming — chardetng or
    equivalent — and transcode to UTF-8.
  • Fail loudly when the bytes decode to nothing plausible, rather than
    writing mojibake into the library. A subtitle that cannot be decoded is a
    failed attempt (#186 already records a failure reason).

Sidecars on disk are UTF-8. That is what §15's "audit by ls" property is
worth nothing without: a file nobody can read is not a satisfied want.

Found while implementing #190.

Depends on: #184

`Provider::download` returns `Fetched.content` as `Vec<u8>` — raw bytes, deliberately, because providers serve Latin-1 and Windows-1252 files and guessing wrong at the provider boundary is worse than handing the bytes on (#184). `arr_subs::srt::parse` deliberately takes `&str` (#190). Nothing in the milestone closes that gap. Every downstream consumer needs text: translation parses cues, `alass` needs a file it can read (#194), and the sidecar written into the title folder has to be in a known encoding (#195). Decode once, in `arr-subs`, between download and everything else: - Honour a UTF-8 BOM when present. - Otherwise detect the encoding rather than assuming — `chardetng` or equivalent — and transcode to UTF-8. - Fail loudly when the bytes decode to nothing plausible, rather than writing mojibake into the library. A subtitle that cannot be decoded is a failed attempt (#186 already records a failure reason). Sidecars on disk are UTF-8. That is what §15's "audit by `ls`" property is worth nothing without: a file nobody can read is not a satisfied want. Found while implementing #190. Depends on: #184
naps62-yolo added this to the Subtitles milestone 2026-08-24 22:36:47 +01:00
naps62-yolo added the area/subsphase/9-subtitlestype/featuredifficulty/moderate labels 2026-08-24 22:36:47 +01:00
naps62-yolo added a new dependency 2026-08-24 22:37:11 +01:00
#196 Close the subtitle gap in the reconcile loop
naps62-yolo added a new dependency 2026-08-24 22:37:27 +01:00
#194 Sync subtitles with alass
naps62-yolo added a new dependency 2026-08-24 22:37:27 +01:00
#195 Write subtitle sidecars into the title folder
naps62-yolo added a new dependency 2026-08-24 22:37:28 +01:00
#213 Convert non-SRT downloads to SRT
naps62-yolo commented 2026-08-24 22:45:01 +01:00
Author
Owner
Copy Link
Copy Source

Implemented: Fetched::decode() -> Result<String> in arr-subs (new decode module). It sniffs a UTF-8/UTF-16 BOM first via encoding_rs::Encoding::decode, and otherwise detects the encoding with chardetng and transcodes. Returns the new Error::Decode { candidate, detail } when the guessed encoding still can't decode the bytes cleanly, rather than writing mojibake. Added chardetng and encoding_rs as pinned workspace deps.

Branch: subtitles/212-decode-utf8.

just ci passed (531/531 tests, clippy and lint checks clean).

Findings

  • Nothing in the repo yet calls Provider::download or Fetched::decode outside tests — no live caller wired up. Expected: #194 (alass) and #195 (sidecar write) are the ones that will consume this.
Implemented: `Fetched::decode() -> Result<String>` in `arr-subs` (new `decode` module). It sniffs a UTF-8/UTF-16 BOM first via `encoding_rs::Encoding::decode`, and otherwise detects the encoding with `chardetng` and transcodes. Returns the new `Error::Decode { candidate, detail }` when the guessed encoding still can't decode the bytes cleanly, rather than writing mojibake. Added `chardetng` and `encoding_rs` as pinned workspace deps. Branch: `subtitles/212-decode-utf8`. `just ci` passed (531/531 tests, clippy and lint checks clean). ## Findings - Nothing in the repo yet calls `Provider::download` or `Fetched::decode` outside tests — no live caller wired up. Expected: #194 (alass) and #195 (sidecar write) are the ones that will consume this.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/api
arr-api
 area/ci
CI and test harness
 area/compat
arr-compat
 area/core
arr-core
 area/daemon
arr-daemon
 area/db
arr-db
 area/dl
arr-dl
 area/indexer
arr-indexer
 area/infra
workspace and tooling
 area/meta
arr-meta
 area/parse
arr-parse
 area/probe
arr-probe
 area/subs
arr-subs crate: providers, extraction, translation, sync
 area/web
web/
 difficulty/easy
bounded, obvious approach, few files
 difficulty/hard
subtle correctness or cross-cutting
 difficulty/moderate
multiple files, judgement, an interface to design
 difficulty/trivial
one file, mechanical, no design decisions
 phase/1-skeleton
workspace, CI, config, database, empty API and SPA
 phase/2-logic
parsing and policy engine, pure, no network
 phase/3-sourcing
TMDB and Prowlarr, read-only
 phase/4-movies
movies end to end
 phase/5-ui
search, buckets, library views, queues
 phase/6-tv
seasons, episodes, tracking, derived status
 phase/7-people
owner tags and notifications
 phase/8-compat
Jellyseerr shim
 phase/9-subtitles
build order: subtitles, DESIGN.md §15
 type/bug
type/chore
type/feature
type/test
No labels area/subs difficulty/moderate phase/9-subtitles type/feature
Milestone
No items
No Milestone Subtitles
Projects
Clear projects
No projects
Assignees
Clear assignees
hermes (Hermes (agent)) naps62 (Miguel Palhas) naps62-yolo (naps62-yolo (agent))
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Blocks
#194 Sync subtitles with alass
yolo/arr
#195 Write subtitle sidecars into the title folder
yolo/arr
#196 Close the subtitle gap in the reconcile loop
yolo/arr
#213 Convert non-SRT downloads to SRT
yolo/arr
Reference: yolo/arr#212