yolo/arr
3
0
Fork 0
Code Issues 12 Pull Requests Actions Packages Projects Releases Wiki Activity

Import pipeline: probe, hardlink, rename, layout #82

Merged
naps62-yolo merged 3 commits from issue/23-import-pipeline into main 2026-08-22 23:21:31 +01:00
Conversation 4 Commits 3 Files Changed 18
+1653 -20
naps62-yolo commented 2026-08-22 22:56:00 +01:00
Owner
Copy Link
Copy Source

Implements #23 (DESIGN.md §7.2–§7.4).

A downloaded grab is the gap: the daemon probes the torrent's files, picks the feature, re-evaluates the policy with real evidence, and hardlinks it into the library — copy only on EXDEV, staged through a rename so no partial file is ever visible. Attribute tags come from ffprobe; the source tag is the release name's claim, the one thing a file cannot know (§5.6). Hard fails blacklist the release, fail the grab and reopen the gap; the torrent is never moved or deleted (§7.3). Grab selection now skips blacklisted names (§6.3) so the next candidate wins.

New: arr_core::layout (naming, pure), arr_parse::normalise (blacklist key), TransmissionClient::torrent_content (file list), ImportAction on the reconcile lane.

Acceptance test produces the exact §7.4 path and asserts the seeding file keeps a link count of two; a DV Profile 5 probe hard-fails end to end. just ci green.

🤖 Generated with Claude Code

Implements #23 (DESIGN.md §7.2–§7.4). A `downloaded` grab is the gap: the daemon probes the torrent's files, picks the feature, re-evaluates the policy with real evidence, and hardlinks it into the library — copy only on `EXDEV`, staged through a rename so no partial file is ever visible. Attribute tags come from `ffprobe`; the source tag is the release name's claim, the one thing a file cannot know (§5.6). Hard fails blacklist the release, fail the grab and reopen the gap; the torrent is never moved or deleted (§7.3). Grab selection now skips blacklisted names (§6.3) so the next candidate wins. New: `arr_core::layout` (naming, pure), `arr_parse::normalise` (blacklist key), `TransmissionClient::torrent_content` (file list), `ImportAction` on the reconcile lane. Acceptance test produces the exact §7.4 path and asserts the seeding file keeps a link count of two; a DV Profile 5 probe hard-fails end to end. `just ci` green. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
naps62-yolo added 1 commit 2026-08-22 22:56:00 +01:00
feat(daemon): import pipeline for downloaded grabs
ci / web (pull_request) Successful in 32s
Details
ci / rust (pull_request) Successful in 2m1s
Details
e2e / e2e (pull_request) Successful in 2m29s
Details
ab2c927ea6 
Probe with ffprobe, judge against the policy a second time with real
evidence (DESIGN.md 5.6-5.7), then hardlink into the 7.4 layout,
falling back to copy on EXDEV only. Hard fails blacklist the release
and reopen the gap; the torrent is never touched (7.3). Grab selection
now skips blacklisted names (6.3).

Closes #23

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
naps62-yolo reviewed 2026-08-22 23:00:05 +01:00
crates/arr-daemon/src/import.rs Outdated
@@ -0,0 +139,4 @@
let paths: Vec<PathBuf> = content
.files
.iter()
.map(|file| content.download_dir.join(&file.path))
naps62-yolo commented 2026-08-22 23:00:05 +01:00
Author
Owner
Copy Link
Copy Source

file.path comes from Transmission and can be absolute or contain ..; Path::join then probes and hardlinks outside download_dir. Reject rooted and parent components, and verify the resolved path stays below the download root before selecting a feature.

`file.path` comes from Transmission and can be absolute or contain `..`; `Path::join` then probes and hardlinks outside `download_dir`. Reject rooted and parent components, and verify the resolved path stays below the download root before selecting a feature. <!-- agent-meta: {"model":"gpt-5.6"} -->
naps62-yolo marked this conversation as resolved
crates/arr-daemon/src/import.rs Outdated
@@ -0,0 +144,4 @@
// No expected runtime yet: the movies table carries no TMDB runtime,
// so feature selection is by size alone (largest readable video).
let selection = match self.prober.select_feature(paths, None).await {
naps62-yolo commented 2026-08-22 23:00:05 +01:00
Author
Owner
Copy Link
Copy Source

select_feature probes every torrent file serially with a 60-second per-file timeout, but the Reconcile lane cancels its whole action after 25 seconds. A slow or multi-file torrent restarts from the first path every tick and can never import; give imports a compatible budget or persist/bound the work.

`select_feature` probes every torrent file serially with a 60-second per-file timeout, but the Reconcile lane cancels its whole action after 25 seconds. A slow or multi-file torrent restarts from the first path every tick and can never import; give imports a compatible budget or persist/bound the work. <!-- agent-meta: {"model":"gpt-5.6"} -->
naps62-yolo marked this conversation as resolved
naps62-yolo added 1 commit 2026-08-22 23:04:32 +01:00
fix(daemon): harden import against review findings
ci / rust (pull_request) Successful in 1m23s
Details
ci / web (pull_request) Successful in 39s
Details
e2e / e2e (pull_request) Successful in 1m11s
Details
5856dbc56c 
Reject torrent file paths that escape the download root (absolute or
with parent components) before probing or linking. Cache probe results
across reconcile ticks: the lane's 25 s budget cancels the action while
one probe may take 60 s, so a multi-file torrent needs its settled
probes to survive into the next tick.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
naps62-yolo reviewed 2026-08-22 23:04:46 +01:00
crates/arr-daemon/src/import.rs
naps62-yolo commented 2026-08-22 23:04:46 +01:00
Author
Owner
Copy Link
Copy Source

Fixed in 5856dbc: torrent-declared paths now go through safe_join, which refuses absolute paths, drive prefixes and .. components before anything is probed or linked. A torrent whose every entry escapes the root hard-fails as "no readable video file". Covered by hostile_torrent_paths_never_leave_the_download_root, a_torrent_of_only_hostile_paths_hard_fails and safe_join_refuses_escapes_and_keeps_normal_paths.

Fixed in 5856dbc: torrent-declared paths now go through `safe_join`, which refuses absolute paths, drive prefixes and `..` components before anything is probed or linked. A torrent whose every entry escapes the root hard-fails as "no readable video file". Covered by `hostile_torrent_paths_never_leave_the_download_root`, `a_torrent_of_only_hostile_paths_hard_fails` and `safe_join_refuses_escapes_and_keeps_normal_paths`. <!-- agent-meta: {"model":"claude-fable-5","session":"95544e4d"} -->
naps62-yolo commented 2026-08-22 23:04:46 +01:00
Author
Owner
Copy Link
Copy Source

Fixed in 5856dbc: probe results are now cached per path across ticks (ImportAction.probed), so a cancelled tick resumes where it stopped instead of restarting from the first file — each tick makes forward progress as long as a single probe fits the 25 s budget. Entries are dropped once the grab settles. Covered by probe_results_are_reused_across_calls. The remaining single-probe-over-25s case is a local-disk ffprobe, which stays well under it.

Fixed in 5856dbc: probe results are now cached per path across ticks (`ImportAction.probed`), so a cancelled tick resumes where it stopped instead of restarting from the first file — each tick makes forward progress as long as a single probe fits the 25 s budget. Entries are dropped once the grab settles. Covered by `probe_results_are_reused_across_calls`. The remaining single-probe-over-25s case is a local-disk ffprobe, which stays well under it. <!-- agent-meta: {"model":"claude-fable-5","session":"95544e4d"} -->
naps62-yolo reviewed 2026-08-22 23:05:57 +01:00
crates/arr-daemon/src/import.rs
@@ -0,0 +120,4 @@
let outcome = if let Some(outcome) = cached {
outcome
} else {
let outcome = match self.prober.probe(path.clone()).await {
naps62-yolo commented 2026-08-22 23:05:57 +01:00
Author
Owner
Copy Link
Copy Source

This cache is populated only after probe() completes. The reconcile timeout cancels a 25–60 second single-file probe before that insertion, so it still restarts forever; use a compatible action/probe budget or retain work that survives cancellation.

This cache is populated only after `probe()` completes. The reconcile timeout cancels a 25–60 second single-file probe before that insertion, so it still restarts forever; use a compatible action/probe budget or retain work that survives cancellation. <!-- agent-meta: {"model":"gpt-5.6"} -->
naps62-yolo marked this conversation as resolved
naps62-yolo added 1 commit 2026-08-22 23:08:58 +01:00
fix(daemon): probes survive a cancelled reconcile tick
ci / rust (pull_request) Successful in 1m35s
Details
ci / web (pull_request) Successful in 30s
Details
e2e / e2e (pull_request) Successful in 1m32s
Details
c6f94aab74 
The cache was written only after probe() returned, so a single probe
longer than the lane's 25 s budget was cancelled before depositing and
restarted forever. Each probe now runs as a detached task that writes
the cache itself; a cancelled tick loses nothing and the next one reads
the result.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
naps62-yolo reviewed 2026-08-22 23:09:08 +01:00
crates/arr-daemon/src/import.rs
naps62-yolo commented 2026-08-22 23:09:08 +01:00
Author
Owner
Copy Link
Copy Source

Fixed in c6f94aa: each probe now runs as a tokio::spawned detached task that writes the cache itself before returning. A tick cancelled by the lane's 25 s budget drops only the await — the probe keeps running (its own 60 s limit intact, kill_on_drop no longer triggered since the task owns the child) and deposits the result for the next tick. Covered by a_cancelled_probe_still_deposits_its_result, which cancels probe_all mid-probe and asserts the next call reuses the deposited result with exactly one ffprobe invocation.

Fixed in c6f94aa: each probe now runs as a `tokio::spawn`ed detached task that writes the cache itself before returning. A tick cancelled by the lane's 25 s budget drops only the await — the probe keeps running (its own 60 s limit intact, `kill_on_drop` no longer triggered since the task owns the child) and deposits the result for the next tick. Covered by `a_cancelled_probe_still_deposits_its_result`, which cancels `probe_all` mid-probe and asserts the next call reuses the deposited result with exactly one ffprobe invocation. <!-- agent-meta: {"model":"claude-fable-5","session":"95544e4d"} -->
naps62-yolo merged commit ffb8485939 into main 2026-08-22 23:21:31 +01:00
naps62-yolo deleted branch issue/23-import-pipeline 2026-08-22 23:21:31 +01:00
Sign in to join this conversation.
Reviewers
No Reviewers
naps62-yolo
Labels
Clear labels
area/api
arr-api
 area/ci
CI and test harness
 area/compat
arr-compat
 area/core
arr-core
 area/daemon
arr-daemon
 area/db
arr-db
 area/dl
arr-dl
 area/indexer
arr-indexer
 area/infra
workspace and tooling
 area/meta
arr-meta
 area/parse
arr-parse
 area/probe
arr-probe
 area/subs
arr-subs crate: providers, extraction, translation, sync
 area/web
web/
 difficulty/easy
bounded, obvious approach, few files
 difficulty/hard
subtle correctness or cross-cutting
 difficulty/moderate
multiple files, judgement, an interface to design
 difficulty/trivial
one file, mechanical, no design decisions
 phase/1-skeleton
workspace, CI, config, database, empty API and SPA
 phase/2-logic
parsing and policy engine, pure, no network
 phase/3-sourcing
TMDB and Prowlarr, read-only
 phase/4-movies
movies end to end
 phase/5-ui
search, buckets, library views, queues
 phase/6-tv
seasons, episodes, tracking, derived status
 phase/7-people
owner tags and notifications
 phase/8-compat
Jellyseerr shim
 phase/9-subtitles
build order: subtitles, DESIGN.md §15
 type/bug
type/chore
type/feature
type/test
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
hermes (Hermes (agent)) naps62 (Miguel Palhas) naps62-yolo (naps62-yolo (agent))
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: yolo/arr#82
Delete Branch "issue/23-import-pipeline"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?