From 82f78e567b9ffa026439535086c5ec79ec13cf0a Mon Sep 17 00:00:00 2001 From: xpltd Date: Sun, 22 Mar 2026 00:29:55 -0500 Subject: [PATCH] Remove dev artifacts from repo: planning docs, egg-info, test prompts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removed: - .bg-shell/ (background shell state) - .planning/ (early planning docs, superseded) - DEPLOY-TEST-PROMPT.md (internal testing prompt) - PROJECT.md (root-level duplicate of README) - backend/media_rip.egg-info/ (Python build artifact) - Caddyfile.example (redundant — Caddyfile is the template) Updated .gitignore to prevent re-addition. --- .bg-shell/manifest.json | 1 - .gitignore | 3 + .planning/PROJECT.md | 138 ---- .planning/STATE.md | 43 -- .planning/config.json | 12 - .planning/research/ARCHITECTURE.md | 662 ------------------ .planning/research/FEATURES.md | 273 -------- .planning/research/PITFALLS.md | 358 ---------- .planning/research/STACK.md | 396 ----------- .planning/research/SUMMARY.md | 204 ------ Caddyfile.example | 4 - DEPLOY-TEST-PROMPT.md | 213 ------ PROJECT.md | 478 ------------- backend/media_rip.egg-info/PKG-INFO | 22 - backend/media_rip.egg-info/SOURCES.txt | 47 -- .../media_rip.egg-info/dependency_links.txt | 1 - backend/media_rip.egg-info/requires.txt | 18 - backend/media_rip.egg-info/top_level.txt | 1 - 18 files changed, 3 insertions(+), 2871 deletions(-) delete mode 100644 .bg-shell/manifest.json delete mode 100644 .planning/PROJECT.md delete mode 100644 .planning/STATE.md delete mode 100644 .planning/config.json delete mode 100644 .planning/research/ARCHITECTURE.md delete mode 100644 .planning/research/FEATURES.md delete mode 100644 .planning/research/PITFALLS.md delete mode 100644 .planning/research/STACK.md delete mode 100644 .planning/research/SUMMARY.md delete mode 100644 Caddyfile.example delete mode 100644 DEPLOY-TEST-PROMPT.md delete mode 100644 PROJECT.md delete mode 100644 backend/media_rip.egg-info/PKG-INFO delete mode 100644 backend/media_rip.egg-info/SOURCES.txt delete mode 100644 backend/media_rip.egg-info/dependency_links.txt delete mode 100644 backend/media_rip.egg-info/requires.txt delete mode 100644 backend/media_rip.egg-info/top_level.txt diff --git a/.bg-shell/manifest.json b/.bg-shell/manifest.json deleted file mode 100644 index 0637a08..0000000 --- a/.bg-shell/manifest.json +++ /dev/null @@ -1 +0,0 @@ -[] \ No newline at end of file diff --git a/.gitignore b/.gitignore index 20cd76c..fb9cd76 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,8 @@ # ── AI tooling (not part of the distributed project) ── .gsd/ .claude/ +.bg-shell/ +.planning/ .DS_Store Thumbs.db *.swp @@ -19,6 +21,7 @@ dist/ build/ __pycache__/ *.pyc +*.egg-info/ .venv/ venv/ target/ diff --git a/.planning/PROJECT.md b/.planning/PROJECT.md deleted file mode 100644 index 53b1972..0000000 --- a/.planning/PROJECT.md +++ /dev/null @@ -1,138 +0,0 @@ -# media.rip() - -## What This Is - -A self-hostable, redistributable Docker container providing a web-based yt-dlp frontend for anyone who wants to rip internet content without touching the CLI. Designed for power users to share with trusted friends, for solo self-hosters who want a clean UI over yt-dlp, and for operators deploying shared or internal instances. Ships with a great default experience — cyberpunk theme, isolated sessions, ephemeral downloads, automatic purge — and is fully configurable via a mounted `config.yaml` so operators can reshape it for any use case. - -Not a MeTube fork. A ground-up rebuild that treats theming, session behavior, purge policy, privacy, and health monitoring as first-class concerns. - -## Core Value - -A user can paste any yt-dlp-supported URL, see exactly what they're about to download, and get it — without creating an account, without sending data anywhere, and without knowing what a terminal is. - -## Current Milestone: v1.0 Initial Release - -**Goal:** Ship a fully functional, self-hostable yt-dlp web frontend — downloads, session management, theming, admin auth, and Docker distribution. - -**Target features:** -- Core download engine (URL detection, format selection, real-time progress) -- Session system (cookie-based, isolated by default, export/import) -- Theming (cyberpunk default, dark, light; drop-in custom themes) -- Admin UI with basic login (Sonarr-style) and live config management -- Health, observability, privacy controls -- Docker distribution with CI/CD pipeline -- Post-ship polish phase (defaults tuning) - -## Requirements - -### Validated - -(None yet — ship to validate) - -### Active - -**Downloads** -- [ ] User can submit any yt-dlp-supported URL (video, audio, playlist) -- [ ] URL auto-detection triggers format scraping as soon as a valid URL is detected (no submit required) -- [ ] User sees full list of available formats/quality with clear file type and size info before downloading -- [ ] User can start all items in a queue or start individual items selectively -- [ ] Playlist support: collapsible parent row + child rows, bulk or individual start -- [ ] Download progress shown in real-time via SSE -- [ ] Completed downloads clearly indicated in UI -- [ ] User can filter and sort the download queue - -**Session & Identity** -- [ ] Session cookie (`mrip_session`) auto-created on first visit, 24hr TTL -- [ ] Session persists across browser refresh and reconnects (SSE replays state on reconnect) -- [ ] User can delete their own session and all associated data (downloads, logs, cookies) -- [ ] User can export their session as a portable file (download history, queue state, preferences) -- [ ] User can import a previously exported session to restore their download history — enables basic identity continuity on persistent instances -- [ ] Cookie auth: user can upload a `cookies.txt` file (Netscape format) per-session for authenticated downloads (private/paywalled content) - -**Link Sharing** -- [ ] User can copy the original source URL to clipboard (share-to-rip) -- [ ] Completed downloads are served at a shareable URL so a friend can download the file directly - -**Theming & UI** -- [ ] Three built-in themes: cyberpunk (default), dark, light -- [ ] Theme selection persisted in localStorage -- [ ] Operators can drop theme directories into `/themes` volume — appears in picker without recompile -- [ ] Theme file format is flat, well-commented, and human-readable — a single file per theme with clear variable names and inline docs so anyone (or an AI) can understand and modify it without prior CSS/frontend knowledge -- [ ] Built-in themes serve as learning examples: heavily commented, covering every UI region, showing what each token controls -- [ ] Responsive layout: desktop (sidebar + table) and mobile (bottom tabs + card list) -- [ ] All touch targets minimum 44px on mobile - -**Admin & Configuration** -- [ ] Admin login: username/password protected admin UI (like qBittorrent/Sonarr/Radarr) — no token-in-header, no raw config required; default credentials set at first boot with forced change prompt -- [ ] Admin panel is the primary config surface — session mode, purge policy, filename templates, branding all configurable via UI without touching files -- [ ] `config.yaml` mount still supported as override layer for operators who prefer infra-as-code / GitOps workflows -- [ ] Operator can configure session mode: `isolated` (default) / `shared` / `open` -- [ ] Operator can configure purge: `scheduled` / `manual` / `never`, with TTL for files and logs independently -- [ ] Operator can configure output filename templates globally (source-aware: YouTube, SoundCloud, generic) -- [ ] Admin panel: active sessions, storage usage, manual purge trigger, unsupported URL log download, live config editor -- [ ] Branding overridable: name, tagline, logo - -**Health & Observability** -- [ ] `GET /api/health` returns service status, yt-dlp version, uptime, and key health flags (disk space available, queue depth) -- [ ] Clear structured logging routed internally (no stdout noise) -- [ ] Health flags surfaced in admin panel - -**Privacy & Data Control** -- [ ] Zero automatic outbound telemetry — no analytics, CDN calls, update checks, or beacons -- [ ] All PII (IPs, session IDs, cookie files) included in purge scope when operator enables it -- [ ] Unsupported URL reporting: user-triggered only, logs domain by default (`report_full_url: false` = domain only), zero automatic submission -- [ ] Config option: `reporting.github_issues` opens pre-filled issue (disabled by default) - -**Unsupported URL Reporting** -- [ ] Failed jobs show error + "Report unsupported site" button -- [ ] Report appends structured entry to `/data/unsupported_urls.log` -- [ ] Admin can download the report log via API - -**Distribution & CI/CD** -- [ ] Single multi-stage Docker image: `ghcr.io/xpltd/media-rip` + `docker.io/xpltd/media-rip` -- [ ] Multi-platform: `linux/amd64` + `linux/arm64` -- [ ] CI on PR: lint (ruff, eslint), type-check (vue-tsc), tests (pytest, vitest), Docker build smoke test -- [ ] CD on tag `v*.*.*`: build, push to both registries, generate GitHub Release with changelog -- [ ] `config.yaml` reference documented in README — all fields, defaults, env var overrides explained - -### Out of Scope - -- External API / arr-stack integration (Radarr/Sonarr-style programmatic use) — documented as future milestone, architecture should not block it -- OAuth / user accounts — end-user sessions are anonymous by design; admin auth is basic username/password only, no SSO -- Real-time chat or social features — not core -- Video posts or re-hosting — media.rip downloads, does not transcode or re-serve content at scale -- Mobile native app — web-first - -## Context - -- Inspired by frustration with MeTube's poor customizability — layout, theming, and defaults are hard to change -- Target operators: power users sharing with trusted friends, solo self-hosters, internal team tools -- Privacy-first: trust is the core proposition — users should feel confident their activity isn't being tracked or leaked -- yt-dlp used as library (`import yt_dlp`), not subprocess — gives fine-grained progress hooks and avoids shell injection -- Session cookie approach chosen so users can reconnect after internet drop and resume where they left off -- Cookie auth (cookies.txt upload) enables downloading paywalled/private content without embedding credentials in the app -- A post-ship polish phase is planned: after the core product is working, do a dedicated pass to tune defaults, tighten the out-of-box experience, and make the cyberpunk theme sing - -## Constraints - -- **Tech Stack**: Python 3.12 + FastAPI (backend), Vue 3 + TypeScript + Vite + Pinia (frontend), SQLite via aiosqlite, SSE for real-time, APScheduler for cron tasks -- **Distribution**: Single Docker image, no external runtime dependencies beyond ffmpeg -- **Zero-config**: Must work out of the box with no mounted config — all settings have safe defaults -- **Compatibility**: Must support at minimum all sites MeTube supports at launch - -## Key Decisions - -| Decision | Rationale | Outcome | -|----------|-----------|---------| -| yt-dlp as library, not subprocess | Fine-grained progress hooks, structured error handling, no shell injection surface | — Pending | -| SSE over WebSockets | Simpler, HTTP-native, auto-reconnect built into browser EventSource | — Pending | -| SQLite for job state | Single-file, zero-dependency, sufficient for concurrency needs | — Pending | -| Session isolation as default | Privacy-first default; operators opt into shared/open | — Pending | -| cookies.txt upload (Netscape format) | yt-dlp native support, well-documented browser extension workflow for users | — Pending | -| External API deferred to v2 | Keeps v1 scope manageable; current API surface designed cleanly so future consumers aren't blocked | — Pending | -| Admin UI auth (basic login) over ADMIN_TOKEN | Lowers barrier for non-technical operators; config-via-UI means no docker restarts to change settings; `config.yaml` still supported as override layer | — Pending | -| Session export/import | Enables identity continuity on persistent instances without a real account system; stays anonymous-first by default | — Pending | -| Theme files human-readable + heavily commented | Lowers floor for customization to near zero — anyone with a text editor or AI assistant can retheme without frontend knowledge | — Pending | - ---- -*Last updated: 2026-03-17 — added session export/import, theme file accessibility goals, admin basic auth login, polish phase* diff --git a/.planning/STATE.md b/.planning/STATE.md deleted file mode 100644 index 23bde08..0000000 --- a/.planning/STATE.md +++ /dev/null @@ -1,43 +0,0 @@ -# Project State - -## Project Reference - -See: .planning/PROJECT.md (updated 2026-03-17) - -**Core value:** A user can paste any yt-dlp-supported URL, see exactly what they're about to download, and get it — without creating an account, without sending data anywhere, and without knowing what a terminal is. - -**Current focus:** Milestone v1.0 — defining requirements - -## Current Position - -Phase: Not started (defining requirements) -Plan: — -Status: Defining requirements -Last activity: 2026-03-17 — Milestone v1.0 started - -## Progress - -`[░░░░░░░░░░] 0%` - -## Recent Decisions - -| Decision | Outcome | -|----------|---------| -| Tech Stack | Python 3.12 + FastAPI, Vue 3 + TypeScript + Vite + Pinia, SQLite, SSE, APScheduler | -| Admin UI auth (basic login) over ADMIN_TOKEN | Pending | -| Session export/import | Pending | -| Theme files human-readable + heavily commented | Pending | - -## Pending Todos - -(none) - -## Blockers / Concerns - -(none) - -## Session Continuity - -Last session: 2026-03-17 -Stopped at: Milestone v1.0 started, proceeding to requirements definition -Resume file: none diff --git a/.planning/config.json b/.planning/config.json deleted file mode 100644 index 3117aca..0000000 --- a/.planning/config.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "mode": "yolo", - "depth": "standard", - "parallelization": true, - "commit_docs": true, - "model_profile": "balanced", - "workflow": { - "research": true, - "plan_check": true, - "verifier": true - } -} \ No newline at end of file diff --git a/.planning/research/ARCHITECTURE.md b/.planning/research/ARCHITECTURE.md deleted file mode 100644 index 34d72aa..0000000 --- a/.planning/research/ARCHITECTURE.md +++ /dev/null @@ -1,662 +0,0 @@ -# Architecture Research - -**Domain:** Self-hosted yt-dlp web frontend (Python/FastAPI + Vue 3) -**Researched:** 2026-03-17 -**Confidence:** HIGH (core integration patterns) / MEDIUM (schema shape, theme system) - ---- - -## Standard Architecture - -### System Overview - -``` -┌─────────────────────────────────────────────────────────────────────┐ -│ BROWSER (Vue 3 SPA) │ -│ │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ -│ │ DownloadQ │ │ AdminPanel │ │ ThemePicker │ │ -│ │ (Vue comp) │ │ (Vue comp) │ │ (Vue comp) │ │ -│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ -│ │ │ │ │ -│ ┌──────┴─────────────────┴──────────────────┴──────────────────┐ │ -│ │ Pinia Stores │ │ -│ │ downloads | session | admin | theme | sse-connection │ │ -│ └──────┬────────────────────────────────────────────────────────┘ │ -│ │ REST (fetch) + SSE (EventSource) │ -└─────────┼───────────────────────────────────────────────────────────┘ - │ - │ HTTP (behind nginx in prod) - ▼ -┌─────────────────────────────────────────────────────────────────────┐ -│ FastAPI (Python 3.12) │ -│ │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ -│ │ /api/dl │ │ /api/admin │ │ /api/sse │ │ -│ │ /api/session│ │ (basic auth)│ │ /api/health │ │ -│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ -│ │ │ │ │ -│ ┌──────┴─────────────────┴──────────────────┴──────────────────┐ │ -│ │ Service Layer │ │ -│ │ DownloadService | SessionService | AdminService | SSEBroker │ │ -│ └──────┬─────────────────────────────────────────┬─────────────┘ │ -│ │ │ │ -│ ┌──────┴──────────────┐ ┌───────────┴──────────────┐ │ -│ │ ThreadPool │ │ APScheduler │ │ -│ │ (yt-dlp workers) │ │ (purge cron) │ │ -│ └──────┬──────────────┘ └──────────────────────────┘ │ -│ │ progress_hook → asyncio.Queue → SSEBroker │ -└─────────┼───────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────┐ -│ Persistence Layer │ -│ ┌──────────────────────┐ ┌───────────────────────────────────┐ │ -│ │ SQLite (aiosqlite) │ │ Filesystem │ │ -│ │ jobs, sessions, │ │ /data/downloads/ (output) │ │ -│ │ config, logs │ │ /data/cookies/ (per-session) │ │ -│ └──────────────────────┘ │ /data/unsupported_urls.log │ │ -│ │ /themes/ (custom) │ │ -│ │ config.yaml (override) │ │ -│ └───────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────┘ -``` - -### Component Responsibilities - -| Component | Responsibility | Notes | -|-----------|----------------|-------| -| Vue SPA | All user interaction, queue visualization, SSE state sync | Built to `/app/static/` at image build time, served by FastAPI StaticFiles | -| Pinia `downloads` store | Download job state, optimistic updates, SSE-driven mutations | SSE events are the source of truth; REST is for initial hydration and commands | -| Pinia `sse-connection` store | Manages EventSource lifecycle, reconnect, missed-event replay | Separate store so reconnect logic doesn't pollute download logic | -| FastAPI routers | Route validation, auth middleware, response shaping | Thin — delegates to services | -| `DownloadService` | Orchestrates yt-dlp jobs, manages queue, dispatches progress to SSEBroker | One service, not per-request; holds job registry | -| `SSEBroker` | Per-session asyncio.Queue map; fan-out to all active SSE connections for a session | Singleton; isolates sessions by `session_id` key | -| `SessionService` | Cookie creation/validation, session CRUD, export/import packaging | Owns session identity; no auth — identity only | -| `AdminService` | Config read/write, live reload, session listing, manual purge | Protected by HTTP Basic auth middleware | -| ThreadPoolExecutor | Runs yt-dlp synchronously; progress hooks bridge back to async via `call_soon_threadsafe` | yt-dlp is synchronous and cannot be awaited directly | -| APScheduler `AsyncIOScheduler` | Purge cron job (file TTL, session TTL, log rotation) | Shares event loop with FastAPI; started in lifespan | -| SQLite (aiosqlite) | Job state, session records, config overrides, unsupported URL log | Single file at `/data/mrip.db` | - ---- - -## Key Integration: yt-dlp Progress → SSE - -This is the most architecturally significant path in the system. Getting it wrong causes either blocking the event loop or losing progress events. - -### The Problem - -yt-dlp's `download()` method is **synchronous and blocking**. It calls `progress_hook` callbacks from inside that synchronous thread. FastAPI runs on asyncio. These two worlds must be bridged without: -- Blocking the event loop (which would stall all SSE streams and API requests) -- Using ProcessPoolExecutor (yt-dlp `YoutubeDL` objects contain file handles — not picklable) - -### The Solution: ThreadPoolExecutor + `call_soon_threadsafe` - -``` -yt-dlp thread (sync) asyncio event loop (async) -───────────────────── ─────────────────────────── -run_in_executor(pool, fn) →→→ awaited by DownloadService - progress_hook(d) fires - loop.call_soon_threadsafe( - queue.put_nowait, event →→→ asyncio.Queue receives event - ) ↓ - SSEBroker.publish(session_id, event) - ↓ - EventSourceResponse yields to browser -``` - -**Rule:** Never call `asyncio.Queue.put()` directly from the yt-dlp thread. Always use `loop.call_soon_threadsafe(queue.put_nowait, event)`. This is the only safe bridge from sync threads to the async event loop. - -### Progress Hook Payload - -yt-dlp calls `progress_hook(d)` where `d` is a dict with these fields: - -```python -{ - "status": "downloading" | "finished" | "error", - "filename": str, - "downloaded_bytes": int, - "total_bytes": int | None, # None if unknown - "total_bytes_estimate": int | None, - "speed": float | None, # bytes/sec - "eta": int | None, # seconds - "elapsed": float, - "tmpfilename": str | None, - # "fragment_index", "fragment_count" for HLS/DASH -} -``` - -Normalize this into a typed `ProgressEvent` before putting it on the queue — never send raw yt-dlp dicts to the browser. - ---- - -## Component Boundaries - -### New Components Required (not pre-existing libraries) - -| Component | File | Why It's Its Own Thing | -|-----------|------|------------------------| -| `SSEBroker` | `app/core/sse_broker.py` | Singleton managing per-session queues; must be referenced from both the download worker thread and the SSE endpoint. Lives outside any request lifecycle. | -| `DownloadService` | `app/services/download.py` | Long-lived, holds job registry (`job_id → job_state`), manages ThreadPoolExecutor lifecycle. Not per-request. | -| `SessionMiddleware` (custom) | `app/middleware/session.py` | Auto-creates `mrip_session` UUID cookie on first request; validates on subsequent. Lighter than Starlette's full SessionMiddleware, which signs the entire session dict into the cookie. We only want an opaque ID. | -| `ConfigManager` | `app/core/config.py` | Merges `config.yaml` overrides onto defaults; exposes live-reload API for admin. SQLite holds the mutable copy; `config.yaml` is read-only at start and writes nothing back. | -| `ThemeLoader` | `app/core/theme_loader.py` | Scans `/themes/` volume directory at startup and on admin request; returns manifest of available themes. Does not compile anything — themes are served as static CSS variable files. | -| `PurgeService` | `app/services/purge.py` | Encapsulates purge logic (file TTL, session TTL, log trim). Called by APScheduler cron and by admin manual-trigger endpoint. | -| `SessionExporter` | `app/services/session_export.py` | Serializes session + job history to JSON archive; validates and imports the reverse. | - -### Modified / Wrapped Components - -| Component | Modification | -|-----------|-------------| -| `sse-starlette` `EventSourceResponse` | Used directly; no modification needed | -| `APScheduler` `AsyncIOScheduler` | Wrapped in lifespan startup/shutdown; no subclassing | -| `aiosqlite` | Wrapped in a thin `Database` context manager for connection reuse across requests via FastAPI dependency injection | - ---- - -## Database Schema Shape - -Single SQLite file at `/data/mrip.db`. All tables use `TEXT` UUIDs as primary keys for portability in exports. - -```sql --- Sessions: cookie identity -CREATE TABLE sessions ( - id TEXT PRIMARY KEY, -- UUID, matches mrip_session cookie value - created_at INTEGER NOT NULL, -- unix timestamp - last_seen INTEGER NOT NULL, - mode TEXT NOT NULL DEFAULT 'isolated', - preferences TEXT NOT NULL DEFAULT '{}' -- JSON blob (theme selection, etc.) -); - --- Jobs: one row per download task -CREATE TABLE jobs ( - id TEXT PRIMARY KEY, -- UUID - session_id TEXT NOT NULL REFERENCES sessions(id), - url TEXT NOT NULL, - title TEXT, - format_id TEXT, - status TEXT NOT NULL, -- queued|downloading|finished|error|cancelled - progress_pct REAL DEFAULT 0, - speed_bps REAL, - eta_secs INTEGER, - error_msg TEXT, - output_path TEXT, -- relative to /data/downloads/ - file_size INTEGER, - created_at INTEGER NOT NULL, - started_at INTEGER, - finished_at INTEGER -); - --- Config: mutable settings (admin UI writes here; config.yaml seeds it) -CREATE TABLE config ( - key TEXT PRIMARY KEY, - value TEXT NOT NULL -- JSON-serialized scalar or object -); - --- Unsupported URL log (append-only) -CREATE TABLE unsupported_urls ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - session_id TEXT, - domain TEXT NOT NULL, -- logged domain only (default) - full_url TEXT, -- NULL unless report_full_url=true - error_msg TEXT, - created_at INTEGER NOT NULL -); -``` - -**Indexes needed:** -- `jobs(session_id, status)` — SSE reconnect replay, queue filtering -- `jobs(finished_at)` — purge queries -- `sessions(last_seen)` — session TTL purge - ---- - -## Recommended Project Structure - -``` -media-rip/ -├── backend/ -│ ├── app/ -│ │ ├── main.py # FastAPI app factory, lifespan, middleware -│ │ ├── core/ -│ │ │ ├── config.py # ConfigManager (yaml merge + SQLite live config) -│ │ │ ├── database.py # aiosqlite connection pool + migration runner -│ │ │ ├── sse_broker.py # SSEBroker singleton -│ │ │ └── theme_loader.py # /themes/ scanner -│ │ ├── middleware/ -│ │ │ └── session.py # mrip_session cookie auto-create/validate -│ │ ├── routers/ -│ │ │ ├── downloads.py # POST /api/dl, GET /api/dl/{id}, DELETE -│ │ │ ├── sessions.py # GET/DELETE /api/session, export/import -│ │ │ ├── sse.py # GET /api/sse (EventSourceResponse) -│ │ │ ├── admin.py # /api/admin/* (basic auth protected) -│ │ │ ├── health.py # GET /api/health -│ │ │ └── themes.py # GET /api/themes (manifest) -│ │ ├── services/ -│ │ │ ├── download.py # DownloadService (ThreadPool + job registry) -│ │ │ ├── purge.py # PurgeService -│ │ │ └── session_export.py # SessionExporter -│ │ └── models/ -│ │ ├── job.py # Pydantic models: JobCreate, JobStatus, ProgressEvent -│ │ ├── session.py # SessionRecord, SessionExport -│ │ └── config.py # ConfigSchema -│ ├── tests/ -│ │ ├── test_sse_broker.py -│ │ ├── test_download_service.py -│ │ └── test_session.py -│ ├── alembic/ # DB migrations (keep even for SQLite — schema evolves) -│ └── pyproject.toml -├── frontend/ -│ ├── src/ -│ │ ├── main.ts -│ │ ├── App.vue -│ │ ├── stores/ -│ │ │ ├── downloads.ts # Job state, queue ops -│ │ │ ├── session.ts # Session identity, export/import -│ │ │ ├── sse.ts # EventSource lifecycle + reconnect -│ │ │ ├── admin.ts # Admin state, config editor -│ │ │ └── theme.ts # Active theme, available themes -│ │ ├── components/ -│ │ │ ├── DownloadQueue/ -│ │ │ ├── FormatPicker/ -│ │ │ ├── ProgressBar/ -│ │ │ ├── PlaylistRow/ -│ │ │ └── AdminPanel/ -│ │ ├── composables/ -│ │ │ └── useSSE.ts # Thin wrapper over sse store -│ │ └── themes/ # Built-in theme CSS variable files (embedded in build) -│ │ ├── cyberpunk.css -│ │ ├── dark.css -│ │ └── light.css -│ ├── public/ -│ └── vite.config.ts -├── themes/ # Volume-mounted custom themes (operator drop-in) -│ └── .gitkeep -├── data/ # Volume-mounted runtime data -│ └── .gitkeep -├── Dockerfile -├── docker-compose.yml # For local dev and reference deploy -└── config.yaml.example -``` - -### Structure Rationale - -- **`backend/app/core/`:** Things that live for the full application lifetime (broker, config, DB pool) vs. `services/` which own business logic and can be unit-tested in isolation. -- **`backend/app/middleware/`:** Session cookie logic in middleware means every request gets `request.state.session_id` populated before it hits any router. No per-route cookie reading. -- **`frontend/src/stores/sse.ts`:** SSE lifecycle is isolated from business stores. Downloads store subscribes to SSE store events. This means reconnect logic doesn't leak into job state logic. -- **`themes/` at repo root:** Separate from `frontend/src/themes/` — built-in themes are compiled into the frontend bundle; operator themes are volume-mounted and served dynamically at runtime. - ---- - -## Data Flow: Key Paths - -### Path 1: URL → Download → SSE Progress → Completion - -``` -1. User pastes URL - Browser: URL field onChange → format-probe fetch (GET /api/dl/probe?url=...) - Backend: yt-dlp.extract_info(url, download=False) in ThreadPool → returns formats - Browser: FormatPicker shows options - -2. User selects format, clicks Download - Browser: POST /api/dl {url, format_id, session_id (from cookie)} - Backend: DownloadService.enqueue(job) → creates DB row (status=queued) - returns {job_id} - -3. SSE stream delivers state - Browser: EventSource on /api/sse (session_id from cookie) - SSEBroker has a queue keyed by session_id - Backend: GET /api/sse → EventSourceResponse(async_generator) - generator: while True: event = await queue.get(); yield event - -4. Download worker executes - Backend: ThreadPoolExecutor.submit(run_download, job_id, url, format_id, opts) - Inside thread: - YoutubeDL(opts).download([url]) - progress_hook fires with {status, downloaded_bytes, ...} - → loop.call_soon_threadsafe( - sse_broker.put_nowait, - session_id, - ProgressEvent(job_id, ...) - ) - On finish: DB update (status=finished, output_path=...) - → call_soon_threadsafe sends "finished" event - -5. Browser receives progress events - SSE store receives raw event → dispatches to downloads store - downloads store: jobs[job_id].progress = event.pct - -6. SSE reconnect (browser drop/refresh) - Browser: EventSource auto-reconnects (built-in) - Backend: GET /api/sse → queries DB for all active/recent jobs for this session - Replays current state as synthetic SSE events before entering live queue -``` - -### Path 2: Admin Config Change (live reload) - -``` -Admin UI → POST /api/admin/config {key, value} - → AdminService.set(key, value) → writes to config table in SQLite - → ConfigManager.invalidate_cache() - → next request picks up new value - (No restart required — config is read from DB on each use, not at startup) -``` - -### Path 3: Drop-in Theme Load - -``` -Operator: docker volume mount ./my-theme/ → /themes/my-theme/ - /themes/my-theme/theme.css (CSS custom properties) - /themes/my-theme/meta.json {name, author, preview_color} - -Backend startup: ThemeLoader.scan() → reads /themes/*/meta.json - GET /api/themes → returns [{id, name, author, preview_color, is_builtin}] - GET /themes/{id}/theme.css → FileResponse (volume-served, not compiled) - -Browser: ThemePicker calls /api/themes, shows list - User selects custom theme → swapped to /themes/id/theme.css - (Built-in themes are already in the bundle as CSS files) -``` - -### Path 4: Session Export/Import - -``` -Export: - GET /api/session/export - → SessionExporter.export(session_id) - → queries: session row + all jobs for session - → zips: export.json + any cookies.txt for this session - → returns StreamingResponse (zip file download) - -Import: - POST /api/session/import (multipart, zip file) - → unzip, validate schema version - → create new session (new UUID, import grants new identity) - → insert jobs (status "finished" only — don't replay active downloads) - → return new session cookie (Set-Cookie: mrip_session=new_uuid) -``` - ---- - -## Architectural Patterns - -### Pattern 1: Sync-to-Async Bridge via `call_soon_threadsafe` - -**What:** yt-dlp progress hooks fire synchronously inside a thread. The running event loop must be captured at app startup and used to safely enqueue events without blocking the thread or corrupting the loop. - -**When to use:** Any time synchronous library code in a worker thread needs to communicate back to the asyncio world. - -**Trade-offs:** Simple and correct. The only alternative (running yt-dlp in a subprocess and parsing stdout) is fragile and loses structured error info. - -**Key snippet shape:** -```python -# In app startup — capture the loop once -loop = asyncio.get_event_loop() - -# In progress hook (called from sync thread) -def progress_hook(d: dict) -> None: - event = ProgressEvent.from_yt_dlp(job_id, d) - loop.call_soon_threadsafe(sse_broker.put_nowait, session_id, event) -``` - -### Pattern 2: Per-Session SSE Queue Fan-Out - -**What:** One `asyncio.Queue` per connected SSE client (not per session). Multiple browser tabs from the same session each get their own queue. SSEBroker maintains `session_id → List[Queue]` and fans out to all queues on `publish()`. - -**When to use:** Always. A single global queue would leak events across sessions — a privacy violation that defeats session isolation. - -**Trade-offs:** Queue cleanup requires detecting client disconnect. `sse-starlette`'s `EventSourceResponse` handles this — the generator raises `asyncio.CancelledError` or `GeneratorExit` when the client disconnects, allowing cleanup in a `finally` block. - -### Pattern 3: SSE Replay on Reconnect - -**What:** When a client reconnects to `/api/sse`, the endpoint first emits synthetic events for all current job states from the DB before entering the live queue. This ensures the UI is fully hydrated on reconnect without requiring a separate REST fetch. - -**When to use:** Any SSE endpoint where the client might have missed events during a disconnect. - -**Trade-offs:** Slightly more complex endpoint logic, but eliminates an entire class of "spinner forever after refresh" bugs. - -### Pattern 4: Config Hierarchy (Defaults → YAML → SQLite) - -**What:** Settings have three layers. Built-in defaults are hardcoded in Python. `config.yaml` overrides them at startup (read-only after that). Admin UI writes to the `config` SQLite table, which is the live source of truth at runtime. - -**When to use:** Operator-facing applications that need both infra-as-code (YAML) and live UI config without restart. - -**Trade-offs:** Two sources of truth during initial startup (YAML seeds SQLite on first boot, then SQLite wins). Must document precedence clearly. YAML never reflects what admin UI has changed. - ---- - -## Anti-Patterns - -### Anti-Pattern 1: Running yt-dlp directly in an async def route - -**What people do:** `await asyncio.to_thread(ydl.download, [url])` inside a route handler. - -**Why it's wrong:** `asyncio.to_thread` uses the default executor, which shares a pool with all other blocking calls. More critically, the progress hook fires from inside that thread and has no safe way to reach the SSE queue without a stored event loop reference. This pattern leads to either lost events or `RuntimeError: no running event loop`. - -**Do this instead:** Use `DownloadService` (a singleton with its own dedicated `ThreadPoolExecutor`), capture `asyncio.get_event_loop()` at app startup, and use `call_soon_threadsafe` in the hook. - -### Anti-Pattern 2: Storing session content in the cookie - -**What people do:** Use Starlette's `SessionMiddleware` which signs the entire session dict into the cookie. - -**Why it's wrong:** Session content (job IDs, preferences) grows unboundedly. Signed cookies can be decoded (just not tampered with). Violates the principle that the browser should hold only an opaque identity token. - -**Do this instead:** Store only a UUID in the `mrip_session` cookie. All session state lives in SQLite keyed by that UUID. - -### Anti-Pattern 3: Single global SSE queue for all sessions - -**What people do:** One `asyncio.Queue` app-wide; all SSE consumers read from it. - -**Why it's wrong:** Every client sees every other client's download events. Violates session isolation (the core privacy promise). Also creates thundering-herd wake-ups for unrelated events. - -**Do this instead:** `SSEBroker` maps `session_id → List[asyncio.Queue]`, one queue per live connection. - -### Anti-Pattern 4: Polling the DB for progress updates from SSE endpoint - -**What people do:** SSE endpoint loops with `await asyncio.sleep(0.5)` and queries the DB for job state changes. - -**Why it's wrong:** Generates constant DB load proportional to active connections × poll frequency. Introduces 0-500ms latency on progress events. Doesn't scale. - -**Do this instead:** DownloadService pushes events directly into the SSE queues via `call_soon_threadsafe`. DB is only written for persistence — SSE reads from the queue. - -### Anti-Pattern 5: Volume-mounting themes into the frontend build directory - -**What people do:** Mount custom themes into `/app/static/themes/` and expect Vue to pick them up. - -**Why it's wrong:** The built-in themes are baked into the static bundle at image build time. A volume mount on the same directory would shadow built-in themes and create confusion. - -**Do this instead:** Built-in themes live at `/app/static/builtin-themes/` (baked in). Custom themes live at `/themes/` (volume-mounted). Frontend fetches the manifest from `/api/themes` to know what's available. `GET /themes/{id}/theme.css` is served by FastAPI's `StaticFiles` mount on the volume directory. - ---- - -## Docker Layering Strategy - -### Multi-Stage Build: 3 Stages - -```dockerfile -# Stage 1: Frontend builder (Node) -FROM node:22-alpine AS frontend-builder -WORKDIR /frontend -COPY frontend/package*.json ./ -RUN npm ci -COPY frontend/ . -RUN npm run build -# Output: /frontend/dist/ - -# Stage 2: Python dependency builder -FROM python:3.12-slim AS python-builder -WORKDIR /build -RUN pip install uv -COPY backend/pyproject.toml backend/uv.lock ./ -RUN uv pip install --system --no-cache -r pyproject.toml -# Installs: fastapi, uvicorn, yt-dlp, sse-starlette, aiosqlite, apscheduler, pyyaml, etc. - -# Stage 3: Final runtime image -FROM python:3.12-slim AS runtime -# Install ffmpeg (required by yt-dlp for muxing) -RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg && rm -rf /var/lib/apt/lists/* -# Copy Python packages from builder -COPY --from=python-builder /usr/local/lib/python3.12 /usr/local/lib/python3.12 -COPY --from=python-builder /usr/local/bin /usr/local/bin -# Copy backend source -COPY backend/app /app/app -# Copy built frontend assets into location FastAPI StaticFiles will serve -COPY --from=frontend-builder /frontend/dist /app/static -# Runtime config -WORKDIR /app -ENV MRIP_DATA_DIR=/data -VOLUME ["/data", "/themes"] -EXPOSE 8000 -CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] -``` - -### Layer Cache Optimization - -The stage order matters for cache hit rates during development: - -1. **Frontend builder first:** Node dependencies are the most stable. `package-lock.json` changes rarely. `npm ci` layer is cache-friendly. -2. **Python deps before source:** `pyproject.toml` changes less often than `app/` code. Source copy is always last within each stage. -3. **ffmpeg in a single RUN:** Combine `apt-get update`, install, and `rm -rf /var/lib/apt/lists/*` in one layer to avoid caching a stale package index. - -### Multi-Platform Build (amd64 + arm64) - -```bash -# CI pipeline (GitHub Actions) -docker buildx build \ - --platform linux/amd64,linux/arm64 \ - --tag ghcr.io/xpltd/media-rip:$VERSION \ - --push \ - . -``` - -**Arm64 consideration:** `ffmpeg` from Debian apt supports arm64 natively — no cross-compile needed. yt-dlp is pure Python — no binary concern. The only risk is any Python package with C extensions (e.g., `aiosqlite` → `sqlite3` → system library). `python:3.12-slim` includes `libsqlite3` for both platforms. - -**QEMU vs. native:** GitHub Actions standard runners are amd64. QEMU emulation for arm64 is slow but correct for this stack (no complex native compilation). If build times become painful, use ARM runners (e.g., Blacksmith or self-hosted). - -### FastAPI Serving Static Files (no nginx needed in single container) - -FastAPI's `StaticFiles` mount is sufficient for this use case (single-instance self-hosted tool, not a CDN-scale app): - -```python -from fastapi.staticfiles import StaticFiles - -# Built frontend assets -app.mount("/assets", StaticFiles(directory="/app/static/assets"), name="assets") - -# Volume-mounted custom themes -app.mount("/themes", StaticFiles(directory=os.environ.get("MRIP_THEMES_DIR", "/themes")), name="themes") - -# SPA fallback: any unmatched path returns index.html -@app.get("/{full_path:path}") -async def spa_fallback(full_path: str): - return FileResponse("/app/static/index.html") -``` - -If an operator wants to put nginx in front (for TLS termination, caching), the container works unchanged behind a reverse proxy. - ---- - -## Build Order (Dependency-Respecting) - -Build phases in this order to avoid blocking work: - -``` -Phase 1: Foundation (no dependencies) -├── Database schema + migrations (aiosqlite, alembic init) -├── ConfigManager (pure Python, no DB dependency) -├── SessionMiddleware (cookie only — no DB needed to write it) -└── SSEBroker (pure asyncio.Queue — no yt-dlp, no DB) - -Phase 2: Core Services (depends on Phase 1) -├── DownloadService skeleton (ThreadPool, queue intake, DB writes) -│ └── yt-dlp integration + progress hook bridge to SSEBroker -├── SSE endpoint (depends on SSEBroker from Phase 1) -│ └── With reconnect/replay from DB -└── Session CRUD endpoints (depends on DB + SessionMiddleware) - -Phase 3: Frontend Core (can start after Phase 2 API shape is stable) -├── Pinia sse store + EventSource lifecycle -├── Pinia downloads store (consumes SSE events) -├── DownloadQueue component (URL input → probe → format picker → enqueue) -└── ProgressBar (driven by downloads store) - -Phase 4: Admin + Auth (depends on Phase 2) -├── AdminService (config read/write) -├── Basic auth middleware on /api/admin/* -├── Admin router (sessions, storage, purge trigger, config editor) -└── Admin UI (Vue components) - -Phase 5: Supporting Features (depends on Phases 2-4) -├── Theme system (ThemeLoader + /api/themes + volume serving) -├── PurgeService + APScheduler integration -├── Session export/import -├── cookies.txt upload (per-session) -└── Unsupported URL logging + admin download - -Phase 6: Distribution -├── Dockerfile (multi-stage) -├── docker-compose.yml -├── GitHub Actions CI (lint, type-check, test, Docker smoke) -└── GitHub Actions CD (tag → build + push + release) -``` - -**Critical path:** Phase 1 → Phase 2 (SSEBroker + yt-dlp bridge) → Phase 3 (SSE consumer). The SSE transport must exist before meaningful frontend progress work can be validated end-to-end. - ---- - -## Integration Points - -### External Dependencies - -| Dependency | Integration Pattern | Critical Notes | -|------------|---------------------|----------------| -| yt-dlp | `import yt_dlp` as library, not subprocess | `YoutubeDL` instance created fresh per job inside worker thread. Not shared. Not passed across process boundary. | -| ffmpeg | Installed in Docker image; yt-dlp finds it via `PATH` | Required for muxing video+audio streams. Not directly called by app code. | -| `sse-starlette` (v3.3.3) | `EventSourceResponse(async_generator)` | Handles ping/heartbeat, client disconnect detection. No subclassing needed. | -| `APScheduler` `AsyncIOScheduler` | Started in FastAPI `lifespan` context manager | Use `AsyncIOScheduler` (not `BackgroundScheduler`) to share the event loop. One instance globally. | -| `aiosqlite` | Thin wrapper for connection reuse via FastAPI `Depends` | One connection pool, not per-request connections. WAL mode for concurrent reads. | - -### Internal Boundaries - -| Boundary | Communication | Notes | -|----------|---------------|-------| -| Worker Thread ↔ SSEBroker | `loop.call_soon_threadsafe(broker.put_nowait, ...)` | Only safe async bridge from sync thread | -| SSEBroker ↔ SSE endpoint | `await queue.get()` in async generator | SSEBroker holds the queue; endpoint holds a reference | -| DownloadService ↔ DB | Direct `aiosqlite` calls | Service owns all job table writes | -| Middleware ↔ Routers | `request.state.session_id` | Middleware populates state; routers read it | -| ConfigManager ↔ All Services | Singleton read via dependency injection | No global variable — injected via `Depends(get_config)` | -| ThemeLoader ↔ Volume | Filesystem scan at startup + on-demand re-scan | No file watchers — re-scan is triggered by API call | - ---- - -## Scaling Considerations - -This is a single-instance self-hosted tool. The relevant scaling axis is concurrent downloads per instance, not users. - -| Concern | Practical Limit | Mitigation | -|---------|-----------------|------------| -| Concurrent downloads | ThreadPoolExecutor defaults (min: 1, configurable) | Expose `max_concurrent_downloads` in config. Default 3 is safe for home use. | -| SQLite write contention | WAL mode handles concurrent reads + single writer fine | Enable `PRAGMA journal_mode=WAL` at DB init. No further action needed for this use case. | -| SSE connection count | asyncio handles hundreds of idle connections trivially | Not a practical concern for self-hosted tool | -| Disk space | operator concern | PurgeService + health endpoint disk-free flag address this | -| yt-dlp blocking | Handled by ThreadPool | GIL is released during I/O-heavy yt-dlp work; threads are effective here | - -The architecture should not block a future "external API" milestone. The service layer is already the right boundary: a future v2 API consumer calls `DownloadService.enqueue()` just like the REST endpoint does — no architectural change required. - ---- - -## Sources - -- yt-dlp asyncio + ProcessPoolExecutor issue: https://github.com/yt-dlp/yt-dlp/issues/9487 -- sse-starlette PyPI (v3.3.3, 2026-03-17): https://pypi.org/project/sse-starlette/ -- FastAPI SSE official docs: https://fastapi.tiangolo.com/tutorial/server-sent-events/ -- FastAPI async/threading patterns: https://fastapi.tiangolo.com/async/ -- Docker multi-platform builds: https://docs.docker.com/build/building/multi-platform/ -- Multi-arch GitHub Actions: https://www.blacksmith.sh/blog/building-multi-platform-docker-images-for-arm64-in-github-actions -- FastAPI + aiosqlite pattern: https://sqlspec.dev/examples/frameworks/fastapi/aiosqlite_app.html -- APScheduler + FastAPI lifespan: https://rajansahu713.medium.com/implementing-background-job-scheduling-in-fastapi-with-apscheduler-6f5fdabf3186 -- FastAPI ThreadPool vs run_in_executor: https://sentry.io/answers/fastapi-difference-between-run-in-executor-and-run-in-threadpool/ - ---- -*Architecture research for: media.rip() v1.0 — Python/FastAPI + Vue 3 + yt-dlp + SSE + SQLite + Docker* -*Researched: 2026-03-17* diff --git a/.planning/research/FEATURES.md b/.planning/research/FEATURES.md deleted file mode 100644 index 3dab8e6..0000000 --- a/.planning/research/FEATURES.md +++ /dev/null @@ -1,273 +0,0 @@ -# Feature Research - -**Domain:** yt-dlp web frontend / self-hosted media downloader -**Researched:** 2026-03-17 -**Confidence:** HIGH (core features), MEDIUM (UX patterns), HIGH (competitor gaps) - -## Feature Landscape - -### Table Stakes (Users Expect These) - -Features users assume exist. Missing these = product feels incomplete. - -| Feature | Why Expected | Complexity | Notes | -|---------|--------------|------------|-------| -| URL paste + download | The core primitive — every tool has this | LOW | Must support all yt-dlp-supported sites, not just YouTube | -| Real-time download progress | Users need feedback; "Processing..." with no indicator is dead UX | MEDIUM | MeTube uses WebSocket; we use SSE — both solve this. SSE is simpler and HTTP-native with auto-reconnect | -| Queue view (active + completed) | Users submit multiple URLs; need to track all of them | LOW | MeTube separates active/done lists; unified queue with status is cleaner | -| Format/quality selection | Power users always want control over resolution, codec, ext | MEDIUM | Must show resolution, codec, ext, filesize estimate. yt-dlp returns all fields: height, vcodec, acodec, ext, filesize, fps | -| Playlist support | Playlists are a primary use case for self-hosters | HIGH | Parent + child job model. MeTube treats playlists as flat — collapsible parent/child is a step up | -| Cancel / remove a download | Users make mistakes | LOW | DELETE /api/downloads/{id}; must handle mid-stream cancellation gracefully | -| Persistent queue across refresh | Losing the queue on page refresh is unacceptable | MEDIUM | Requires SSE `init` event replaying state on connect. MeTube uses state file; our SQLite-backed SSE replay is equivalent | -| Mobile-accessible UI | >50% of self-hoster interactions happen on phone or tablet | HIGH | No existing yt-dlp web UI does mobile well. All competitors are desktop-first. 44px touch targets, bottom nav required | -| Docker distribution | The self-hosted audience expects Docker | LOW | Single image, both registries, amd64 + arm64 | -| Health endpoint | Ops audiences rely on this for monitoring integrations (Uptime Kuma, etc.) | LOW | `GET /api/health` with version, uptime, disk space, queue depth | - -### Differentiators (Competitive Advantage) - -Features that set the product apart. Not required, but valued. - -| Feature | Value Proposition | Complexity | Notes | -|---------|-------------------|------------|-------| -| Session isolation (isolated / shared / open modes) | MeTube Issue #591 closed as "won't fix" — maintainer dismisses multi-user isolation as bloat; community forked it to add this | HIGH | Cookie-based httpOnly UUID4; operator chooses mode; addresses the exact pain point that created demand for forks | -| Cookie auth (cookies.txt upload per-session) | Enables paywalled/private content without embedding credentials in the app; yt-dlp Netscape format is well-documented | MEDIUM | Files must be scoped per-session, purged on session clear. Security note: cookie files are sensitive — never log, never expose via API, delete on purge | -| Drop-in custom themes via volume mount | No competitor offers this. MeTube has light/dark/auto only via env var. yt-dlp-web-ui has no theming | HIGH | CSS variable contract required first. Theme directory: theme.css + metadata.json + optional preview.png. Hot-loaded at startup | -| Heavily commented built-in themes as documentation | Lowers floor for customization to near-zero — anyone with a text editor or AI can retheme | LOW | No runtime cost. Every CSS token documented inline. Built-in themes serve as learning examples | -| Admin UI with username/password login (not raw token) | yt-dlp-web-ui uses JWT tokens in headers/query params — not user-friendly. MeTube has no admin UI at all. qBittorrent/Sonarr-style login is the expected self-hosted pattern | MEDIUM | First-boot credential setup with forced change prompt. Config-via-UI means no docker restarts for settings changes | -| Session export/import | No competitor offers portable session state. Enables identity continuity on persistent instances without a real account system | MEDIUM | JSON export of download history + queue state + preferences. Import restores history. Does not require sign-in, stays anonymous-first | -| Unsupported URL reporting with audit log | No competitor surfaces extraction errors with actionable reporting. MeTube just shows "error" | LOW | User-triggered only. Logs domain by default. Admin downloads log. Optional GitHub issue prefill | -| Source-aware output templates | Sensible per-site defaults (YouTube: uploader/title, SoundCloud: uploader/title, generic: title). MeTube uses one global template | LOW | Config-driven. Per-download override also supported | -| Link sharing (completed file URL) | Users want to share a ripped file with a friend — a direct download URL removes the "now what?" question | LOW | Serve completed files under predictable path. Requires knowing the output filename | -| Zero automatic outbound telemetry | Competing tools have subtle CDN calls, Google Fonts, or update checks. Trust is the core proposition | LOW | No external requests from container. All fonts/assets bundled or self-hosted | -| Cyberpunk default theme | Visual identity differentiator. Every other tool ships with plain material/tailwind defaults | MEDIUM | #00a8ff/#ff6b2b, JetBrains Mono, scanlines, grid overlay. Makes first impressions memorable | - -### Anti-Features (Commonly Requested, Often Problematic) - -Features that seem good but create problems. - -| Feature | Why Requested | Why Problematic | Alternative | -|---------|---------------|-----------------|-------------| -| OAuth / SSO integration | Multi-user deployments want centralized auth | Massive scope increase; introduces external runtime dependency; anonymous-first identity model conflicts with account-based auth | Reverse proxy handles AuthN (Authentik, Authelia, Traefik ForwardAuth); media.rip handles AuthZ via session mode + admin token | -| Real-time everything via WebSocket | Seems more capable than SSE | WebSockets require persistent bidirectional connections, more complex infra, harder to load-balance; SSE covers 100% of the UI's actual needs (server-push only) | SSE — simpler, HTTP-native, auto-reconnecting via browser EventSource | -| User accounts / registration | Makes multi-user feel "proper" | Adds password hashing, email, account management, password reset flow — massive scope for a download tool; users expect anonymous operation | Session isolation mode: each browser gets its own cookie-scoped queue without any account | -| Automatic yt-dlp update on startup | Ensures latest extractor support | Breaks immutable containers and reproducible builds; version drift between deployments; network dependency at boot time | Pin yt-dlp version in requirements.txt; publish new image on yt-dlp releases via CI | -| Embedded video player | Looks impressive in demos | Adds significant frontend complexity, licensing surface for codecs, and scope creep for a downloader tool; most files need to go to Jellyfin/Plex anyway | Serve files at predictable paths; let users open in their preferred player | -| Telegram / Discord bot integration | Power users want remote submission | Separate runtime concern; adds credentials management, API rate limits, message parsing complexity; not what v1 needs to prove | Documented as v2+ extension point; clean API surface makes it straightforward to add later | -| Subscription / channel monitoring | "Set it and forget it" appeal | Fundamentally different product — a scheduler/archiver vs a download UI; scope would double; tools like Pinchflat, TubeArchivist do this better | Out of scope — architecture should not block adding it; APScheduler is already present for purge | -| Per-format download presets | Advanced users want "my 720p MP3 preset" saved | Medium complexity, but defers well to v1.x — v1 needs live format selection working first before persisting preferences | Implement after session system is stable; presets can be stored per-session in config | -| FlareSolverr / Cloudflare bypass | Some sites block yt-dlp | Introduces external service dependency, legal gray area, maintenance surface; YTPTube does this but it's an edge case | cookies.txt upload solves the authenticated content problem for most users; FlareSolverr is too niche for v1 | - -## Feature Dependencies - -``` -[SQLite Job Store] - └──required-by──> [Download Queue View] - └──required-by──> [Real-Time SSE Progress] - └──required-by──> [Playlist Parent/Child Jobs] - -[Session System (cookie-based)] - └──required-by──> [Session Isolation Mode] - └──required-by──> [Cookie Auth (cookies.txt per-session)] - └──required-by──> [Session Export/Import] - └──required-by──> [SSE per-session stream] - -[SSE Bus (per-session)] - └──required-by──> [Real-Time Progress Updates] - └──required-by──> [Init replay on reconnect] - └──required-by──> [purge_complete event] - -[yt-dlp Integration (library mode)] - └──required-by──> [Format/Quality Selection (GET /api/formats)] - └──required-by──> [Download execution] - └──required-by──> [Playlist resolution → child jobs] - └──required-by──> [Error detection → unsupported URL reporting] - -[Admin Auth (username/password)] - └──required-by──> [Admin Panel UI] - └──required-by──> [Purge API endpoint] - └──required-by──> [Session list / storage endpoints] - └──required-by──> [Unsupported URL log download] - -[CSS Variable Contract (base.css)] - └──required-by──> [Built-in themes (cyberpunk, dark, light)] - └──required-by──> [Drop-in custom themes] - └──required-by──> [Theme picker UI] - -[Theme Picker UI] - └──enhances──> [Drop-in custom themes] - -[Completed Download File Serving] - └──required-by──> [Link sharing (shareable download URL)] - -[Purge Scheduler (APScheduler)] - └──enhances──> [Session TTL expiry] - └──enhances──> [File and log TTL purge] - -[Format/Quality Selection] - └──enhances──> [Per-download output template override] - -[Session Export] - └──requires──> [Session System] - └──conflicts-with~~> [open mode] (no session = nothing to export) -``` - -### Dependency Notes - -- **Session system required before session export/import:** No session state to serialize without it. Export is meaningless in `open` mode. -- **SSE bus must exist before progress updates:** Progress hooks from yt-dlp thread pool need a dispatcher to push events to the correct session's queue. -- **yt-dlp integration required before format selection:** `GET /api/formats?url=` calls `yt-dlp.extract_info(process=False)` — format list is live-extracted, not pre-cached. -- **CSS variable contract required before any theming:** All three built-in themes and the drop-in theme system depend on the base.css token contract being stable. Changing token names later breaks all custom themes operators have written. -- **Job store required before queue view:** The frontend queue is a projection of SQLite state replayed via SSE `init` events — the DB is the source of truth, not frontend memory. -- **Admin auth required before admin panel:** Admin routes must be protected before the panel is built, otherwise the panel ships with no auth and operators have no safe path to production. -- **File serving endpoint required before link sharing:** Shareable URLs point to a served file path. This is a FastAPI `StaticFiles` or explicit route serving `/downloads`. - -## MVP Definition - -### Launch With (v1.0) - -Minimum viable product — the full target feature set per PROJECT.md. - -- [x] URL submission + auto-detection triggers format scraping — core primitive -- [x] Format/quality selector (populated live from yt-dlp info extraction) — power users won't use a tool that hides quality choice -- [x] Real-time progress via SSE (queued → extracting → downloading → completed/failed) — no progress = no trust -- [x] Download queue: filter, sort, cancel, playlist collapsible parent/child — queue management is table stakes -- [x] Session system: isolated (default) / shared / open — the primary differentiation from MeTube; isolated mode is the zero-config safe default -- [x] SSE init replay on reconnect — required for page refresh resilience; without this isolated mode is useless -- [x] Cookie auth (cookies.txt upload per-session, Netscape format) — enables paywalled content; the practical reason people move off MeTube -- [x] Purge system: scheduled / manual / never; independent file + log TTL — ephemeral storage is the contract with users -- [x] Three built-in themes: cyberpunk (default), dark, light — visual identity and immediate differentiation -- [x] Drop-in custom theme system (volume mount) — the feature request MeTube refuses to build -- [x] Mobile-responsive layout (bottom tabs + card list at <768px) — no competitor does mobile; 44px touch targets -- [x] Admin panel: username/password login, session list, storage, manual purge, unsupported URL log, live config — operators need a UI, not raw config -- [x] Unsupported URL reporting (user-triggered, domain-only by default) — trust feature; users see exactly what gets logged -- [x] Health endpoint (`GET /api/health`) — Uptime Kuma and similar monitoring tools are table stakes for self-hosters -- [x] Session export/import — enables identity continuity on persistent instances -- [x] Link sharing (source URL clipboard + completed file shareable URL) — reduces friction for the "share with a friend" use case -- [x] Zero automatic outbound telemetry — non-negotiable privacy baseline -- [x] Docker: single image, GHCR + Docker Hub, amd64 + arm64 — distribution is a feature - -### Add After Validation (v1.x) - -Features to add once core is working and v1.0 is shipped. - -- [ ] Per-format/quality download presets — add when session system is stable and users ask for it -- [ ] Branding polish pass — tune cyberpunk defaults, tighten out-of-box experience, ensure built-in theme comments are comprehensive -- [ ] `reporting.github_issues: true` — pre-filled GitHub issue opening; disabled by default, enable only after log download is validated -- [ ] Queue filter/sort persistence — store last sort state in localStorage - -### Future Consideration (v2+) - -Features to defer until product-market fit is established. - -- [ ] External arr-stack API (Radarr/Sonarr programmatic integration) — architecture designed not to block this; clean API surface ready -- [ ] Download presets / saved quality profiles — needs session stability first -- [ ] Subscription / channel monitoring — fundamentally different product scope; defer to TubeArchivist/Pinchflat integration or separate milestone -- [ ] Telegram/Discord bot — documented extension point; clean REST API makes it straightforward - -## Feature Prioritization Matrix - -| Feature | User Value | Implementation Cost | Priority | -|---------|------------|---------------------|----------| -| URL submission + download | HIGH | LOW | P1 | -| Real-time SSE progress | HIGH | MEDIUM | P1 | -| Format/quality selector | HIGH | MEDIUM | P1 | -| Job queue (view + cancel) | HIGH | LOW | P1 | -| Playlist parent/child jobs | HIGH | HIGH | P1 | -| Session isolation (cookie-based) | HIGH | HIGH | P1 | -| SSE init replay on reconnect | HIGH | MEDIUM | P1 | -| Three built-in themes | HIGH | MEDIUM | P1 | -| Mobile-responsive layout | HIGH | HIGH | P1 | -| Docker distribution | HIGH | LOW | P1 | -| Health endpoint | MEDIUM | LOW | P1 | -| Cookie auth (cookies.txt upload) | HIGH | MEDIUM | P1 | -| Purge system (scheduled/manual/never) | MEDIUM | MEDIUM | P1 | -| Admin panel (username/password) | MEDIUM | HIGH | P1 | -| Drop-in custom themes (volume mount) | MEDIUM | HIGH | P1 | -| Session export/import | MEDIUM | MEDIUM | P1 | -| Unsupported URL reporting | LOW | LOW | P1 | -| Link sharing | LOW | LOW | P1 | -| Zero outbound telemetry | HIGH | LOW | P1 (constraint, not feature) | -| Source-aware output templates | MEDIUM | LOW | P1 | -| Per-format download presets | MEDIUM | MEDIUM | P2 | -| GitHub issue prefill for reporting | LOW | LOW | P2 | -| Subscription/channel monitoring | MEDIUM | HIGH | P3 | -| Arr-stack API integration | MEDIUM | HIGH | P3 | - -**Priority key:** -- P1: Must have for v1.0 launch -- P2: Should have in v1.x -- P3: Future milestone - -## Competitor Feature Analysis - -| Feature | MeTube | yt-dlp-web-ui | ytptube | media.rip() | -|---------|--------|---------------|---------|-------------| -| URL submission | Yes | Yes | Yes | Yes | -| Real-time progress | WebSocket | WebSocket/RPC | WebSocket | SSE (simpler, auto-reconnect) | -| Format selection | Quality presets (no live extraction) | Yes | Yes (presets) | Live extraction via `GET /api/formats` | -| Playlist support | Yes (flat) | Yes | Yes | Yes (collapsible parent/child) | -| Session isolation | No — all sessions see all downloads (closed as won't fix) | No | Basic auth only | Yes — isolated/shared/open modes | -| Cookie auth | Yes (global, not per-session) | No | Yes | Yes (per-session, purge-scoped) | -| Theming | light/dark/auto env var | None | None | 3 built-ins + drop-in custom themes | -| Mobile-first UI | No (desktop-first) | No | No | Yes (bottom tabs, card list, 44px targets) | -| Admin panel | No | Basic auth header | Basic auth | Username/password login UI, config editor | -| Session export/import | No | No | No | Yes | -| Purge policy | `CLEAR_COMPLETED_AFTER` only | No | No | scheduled/manual/never, independent TTLs | -| Unsupported URL reporting | Error shown only | Error shown only | Error shown only | User-triggered log + admin download | -| Health endpoint | No | No | No | Yes — version, uptime, disk space, queue depth | -| Link sharing | Base URL config only | No | No | Clipboard + direct file download URL | -| Zero telemetry | Yes | Yes | Yes | Yes (explicit design constraint) | -| Docker distribution | Yes (amd64 only) | Yes | Yes | Yes (amd64 + arm64) | - -## Edge Cases and Expected Behaviors - -### Format Selection - -- **Slow info extraction:** `GET /api/formats?url=` calls `extract_info(process=False)` — for some sites this takes 3-10 seconds. UI must show a loading state on the format picker immediately after URL is pasted. -- **No formats returned:** Some sites return a direct URL without format list. UI should fall back to "Best available" option gracefully. -- **Audio-only formats:** Some formats have `vcodec: none` — these should be labeled clearly (e.g., "Audio only — MP3 128kbps"). -- **Format IDs are extractor-specific:** `format_id` values are not portable across sites; always pass them as opaque strings to yt-dlp. -- **filesize field is frequently null:** Many formats don't report filesize in the info_dict. Show "~estimate" or "unknown" — never show 0. - -### Cookie Auth - -- **Cookie expiry:** Cookies expire within ~2 weeks of export. yt-dlp will fail with auth error after expiry — job should show `failed` with a "cookies may be expired" hint. -- **Cookie scope:** cookies.txt contains all site cookies from the browser export. Users should understand this is sensitive. Never log cookie file contents; purge on session clear. -- **Chrome cookie extraction broken since July 2024:** Chrome's App-Bound Encryption makes external extraction impossible. Firefox is the recommended browser for cookie export. UI should surface this note in the cookie upload flow. -- **CRLF vs LF:** Windows-generated cookies.txt files may use CRLF line endings, causing yt-dlp parse errors. Backend should normalize to LF on upload. - -### Playlist Downloads - -- **Large playlists:** A 200-video playlist creates 201 rows in the queue (1 parent + 200 children). UI must handle this gracefully — collapsed by default, with count shown on parent row. -- **Mixed success/failure in playlists:** Some child videos in a playlist may be geo-blocked or removed. Parent job should complete with a `partial` status or show child failure counts. -- **Playlist URL re-extraction:** If a user submits the same playlist URL twice, they get two independent parent jobs (keyed by UUID, not URL). This is intentional per PROJECT.md. - -### Session System - -- **SSE reconnect race:** If the user refreshes while a download is mid-progress, the SSE `init` event must replay the current job state. Without this, the queue appears empty after refresh even though downloads are running. -- **Session mode changes by operator:** If an operator switches from `isolated` to `shared` mid-deployment, existing per-session rows remain scoped to their session IDs. `shared` mode queries all rows regardless of session_id. This is a data model concern — no migration needed, but operator docs should explain the behavior. -- **`open` mode + session export conflict:** In `open` mode, no session is assigned (session_id = null). Session export has nothing to export. UI should hide the export button in `open` mode. - -### Purge - -- **Purge while download is active:** Purge must skip jobs with status `downloading` or `queued`. Only `completed`, `failed`, and `expired` jobs are eligible. -- **File already deleted manually:** If a user deletes a file from `/downloads` outside the app, purge should handle the missing file gracefully (log it, continue). -- **Log TTL vs file TTL independence:** The design intentionally allows keeping logs longer than files (e.g., files_ttl_hours: 24, logs_ttl_hours: 168). The purge.scope config controls what gets deleted. - -## Sources - -- [MeTube GitHub — alexta69/metube](https://github.com/alexta69/metube) -- [MeTube Issue #591 — User management / per-user isolation request](https://github.com/alexta69/metube/issues/591) -- [MeTube Issue #535 — Optional login page request](https://github.com/alexta69/metube/issues/535) -- [yt-dlp-web-ui — marcopiovanello/yt-dlp-web-ui](https://github.com/marcopiovanello/yt-dlp-web-ui) -- [yt-dlp-web-ui Authentication methods wiki](https://github.com/marcopiovanello/yt-dlp-web-ui/wiki/Authentication-methods) -- [ytptube — arabcoders/ytptube](https://github.com/arabcoders/ytptube) -- [yt-dlp Information Extraction Pipeline — DeepWiki](https://deepwiki.com/yt-dlp/yt-dlp/2.2-information-extraction-pipeline) -- [yt-dlp cookie system — DeepWiki](https://deepwiki.com/yt-dlp/yt-dlp/5.5-browser-integration-and-cookie-system) -- [The Ultimate Guide to GUI Front-Ends for yt-dlp 2025 — BrightCoding](https://www.blog.brightcoding.dev/2025/12/06/the-ultimate-guide-to-gui-front-ends-for-youtube-dl-yt-dlp-download-videos-like-a-pro-2025-edition/) -- [6 Ways to Get YouTube Cookies for yt-dlp in 2026 — DEV Community](https://dev.to/osovsky/6-ways-to-get-youtube-cookies-for-yt-dlp-in-2026-only-1-works-2cnb) -- [MeTube on Hacker News — user discussion of limitations](https://news.ycombinator.com/item?id=41098974) - ---- -*Feature research for: yt-dlp web frontend / self-hosted media downloader* -*Researched: 2026-03-17* diff --git a/.planning/research/PITFALLS.md b/.planning/research/PITFALLS.md deleted file mode 100644 index 8f703a4..0000000 --- a/.planning/research/PITFALLS.md +++ /dev/null @@ -1,358 +0,0 @@ -# Pitfalls Research - -**Domain:** yt-dlp web frontend — FastAPI + Vue 3 + SSE + SQLite + Docker -**Researched:** 2026-03-17 -**Confidence:** HIGH (critical pitfalls verified via official yt-dlp issues, sse-starlette docs, CVE advisories; MEDIUM for performance traps and Docker sizing which rely on community sources) - ---- - -## Critical Pitfalls - -### Pitfall 1: Using a Single YoutubeDL Instance for Concurrent Downloads - -**What goes wrong:** -Multiple in-flight downloads share one `YoutubeDL` instance. Instance state (cookies, temp files, internal logger, download archive state) is mutated per-download, causing downloads to corrupt each other's progress data, swap cookies, or raise `TypeError` on `None` fields when hooks fire out of order. - -**Why it happens:** -yt-dlp is documented as a library by example (`with YoutubeDL(opts) as ydl: ydl.download([url])`), which looks reusable. There is no explicit "not thread-safe" warning in the README. Developers assume the object is stateless between calls. - -**How to avoid:** -Create a fresh `YoutubeDL` instance per download job, inside the worker function. Never share an instance across concurrent threads or tasks: - -```python -def _run_download(job_id: str, url: str, opts: dict): - with YoutubeDL({**opts, "progress_hooks": [make_hook(job_id)]}) as ydl: - ydl.download([url]) -``` - -Run this inside `loop.run_in_executor(thread_pool, _run_download, ...)` so the FastAPI event loop is not blocked. The YoutubeDL object never crosses the thread boundary. - -**Warning signs:** -- Progress percentages jump between unrelated jobs -- Two downloads finish at the same time and one reports 0% or corrupted size -- `TypeError: '>' not supported between 'NoneType' and 'int'` in progress hook (a known issue when hook receives stale None from another job's state) - -**Phase to address:** -Core download engine (Phase 1 / foundation). This is the fundamental architecture decision — get it right before building progress reporting on top of it. - ---- - -### Pitfall 2: Calling asyncio Primitives from a yt-dlp Progress Hook - -**What goes wrong:** -The progress hook fires inside the `ThreadPoolExecutor` worker thread, not on the asyncio event loop. Calling `asyncio.Queue.put()`, `asyncio.Event.set()`, or any awaitable directly from the hook raises `RuntimeError: no running event loop` or silently does nothing. - -**Why it happens:** -Progress hooks feel like callbacks, and callbacks in async Python code are usually called on the event loop. But yt-dlp is synchronous — its hooks fire on whichever OS thread is running the download. `loop.run_in_executor` moves the whole call to a thread pool; the hook fires inside that thread. - -**How to avoid:** -Use `loop.call_soon_threadsafe()` to bridge the thread back to the event loop: - -```python -def make_hook(job_id: str, loop: asyncio.AbstractEventLoop, queue: asyncio.Queue): - def hook(d: dict): - # Called from thread — must not await or call asyncio directly - loop.call_soon_threadsafe(queue.put_nowait, { - "job_id": job_id, - "status": d.get("status"), - "downloaded": d.get("downloaded_bytes"), - "total": d.get("total_bytes"), - }) - return hook -``` - -Capture `asyncio.get_event_loop()` in the FastAPI startup context (before executor threads start) and pass it into the hook factory. - -**Warning signs:** -- SSE stream connects but never receives progress updates -- `RuntimeError: no running event loop` in thread worker logs -- Progress updates arrive in large batches rather than incrementally (queued but not flushed) - -**Phase to address:** -Core download engine (Phase 1). The hook bridging must be wired before SSE progress streaming is built. - ---- - -### Pitfall 3: SSE Connection Leak from Swallowed CancelledError - -**What goes wrong:** -When a client disconnects, `sse-starlette` raises `asyncio.CancelledError` in the generator coroutine. If the generator catches it without re-raising (common in `try/except Exception` blocks), the task group never terminates: the ping task, the disconnect listener, and the downstream SSE write loop all become zombie tasks. Over time, the server accumulates connection handles, event queues, and memory. - -**Why it happens:** -`except Exception` catches `CancelledError` in Python 3.7 (it inherits from `BaseException` as of 3.8, but code written for 3.7 patterns is still common). Developers add broad exception handlers to "safely" clean up resources, not realizing they're suppressing the cancellation signal. - -**How to avoid:** -Always use `try/finally` for cleanup and never use bare `except Exception` around SSE generator bodies: - -```python -async def event_generator(request: Request, session_id: str): - try: - async for event in _stream_events(session_id): - if await request.is_disconnected(): - break - yield event - except asyncio.CancelledError: - # Clean up queues, unsubscribe session - _cleanup_session_stream(session_id) - raise # ALWAYS re-raise - finally: - _cleanup_session_stream(session_id) -``` - -**Warning signs:** -- Server memory grows slowly over time even with low active user count -- `asyncio.all_tasks()` shows growing number of `sse_starlette` tasks -- CPU spikes at idle as zombie ping tasks fire continuously - -**Phase to address:** -SSE streaming (Phase 2). Must be enforced before load testing; the leak is invisible at low connection counts and only surfaces under sustained use. - ---- - -### Pitfall 4: Purge Job Deleting Files for Active Downloads - -**What goes wrong:** -The APScheduler purge job queries jobs older than TTL and deletes their files. If a download is actively writing to disk when the purge runs, the file is deleted mid-write. The download worker then fails with `FileNotFoundError` or produces a zero-byte file. The job status in SQLite may be stuck in `downloading` forever. - -**Why it happens:** -Purge logic typically queries by `created_at < now() - TTL` or `completed_at < now() - TTL`. If `completed_at` is NULL for an active download, range logic can accidentally include it depending on NULL handling in the SQL query. Additionally, "complete" status transitions may lag: a job is marked `completed` in the DB a moment after the file is fully written, leaving a window. - -**How to avoid:** -Add an explicit `status != 'downloading'` filter to every purge query — never rely on timestamp alone: - -```sql -DELETE FROM jobs -WHERE status IN ('completed', 'failed', 'cancelled') - AND completed_at < :cutoff_ts -``` - -Also: before deleting a file path, verify the corresponding job row has a terminal status. Write a test that starts a slow download (sleep in a test hook) and triggers purge mid-download — verify the file is not touched. - -**Warning signs:** -- Downloads succeed in tests but randomly fail in production under load -- Jobs stuck in `downloading` status in DB with no active worker -- Zero-byte files in the download directory - -**Phase to address:** -Purge/session management (Phase 3). Write the status-guard test as part of the purge implementation, not after. - ---- - -### Pitfall 5: SSE Reconnect Storm on Page Reload - -**What goes wrong:** -When `EventSource` loses connection (server restart, tab backgrounded, network blip), the browser immediately retries every 3 seconds by default. If the frontend does not track `Last-Event-ID` and the server does not replay recent events, every reconnect gets a blank slate — the UI shows empty progress or "unknown" status for all in-progress downloads. Users refresh repeatedly, multiplying connections. On slow networks, multiple tabs from the same session each open their own SSE connection, exhausting the 6-connection-per-domain HTTP/1.1 limit. - -**Why it happens:** -SSE reconnect is automatic and invisible — developers build the happy path but don't test what happens after a reconnect. `Last-Event-ID` support requires the server to track sent event IDs and replay them, which is non-trivial to implement late. - -**How to avoid:** -- Assign an incrementing `event_id` to every SSE message from day one (can be a job-scoped counter or a global sequence). -- On reconnect, read `Last-Event-ID` header and replay all events for the session that occurred after that ID. -- Replay only the current state snapshot (latest status per job), not the full event log — prevents replay storms. -- Set `retry: 5000` in the SSE stream to slow down reconnect attempts. -- Use HTTP/2 in the Docker container (serve via `uvicorn --http h2` or behind nginx/caddy) to lift the 6-connection limit. - -**Warning signs:** -- After page reload, download cards show "Unknown" or empty progress -- Browser devtools Network tab shows rapid repeated connections to `/api/events` -- Multiple tabs stop receiving updates (one tab's connection blocks others on HTTP/1.1) - -**Phase to address:** -SSE streaming (Phase 2). Must be designed in from the start — adding `Last-Event-ID` replay retroactively requires event log storage. - ---- - -### Pitfall 6: cookies.txt File Leakage via Redirect Attack (CVE-2023-35934) - -**What goes wrong:** -yt-dlp passes uploaded cookies as a `Cookie` header to the file downloader for every request, including redirects. A malicious URL can redirect to an attacker-controlled host, leaking the user's session cookies for the original site. In a multi-user deployment, one user's cookies for YouTube, Vimeo, or Patreon are sent to any host that redirects the download. - -**Why it happens:** -yt-dlp versions before 2023-07-06 do not scope cookies to the origin domain at the file download stage. The CVE affects youtube-dl (all versions) and all yt-dlp versions before the fix. The attack requires no exploit — it is the normal redirect behavior, just exploited. - -**How to avoid:** -- Pin yt-dlp to >= 2023-07-06 (the patched version). Verify in `requirements.txt` and Docker build. -- Store cookies.txt files with per-session isolation: `data/sessions/{session_id}/cookies.txt` — never share files across sessions. -- Delete cookies.txt after the download job completes (or on session purge) so they do not persist on disk. -- Never log the cookies.txt path in any publicly readable log. -- In the security model: treat uploaded cookies as highly sensitive credentials, equivalent to a login token. - -**Warning signs:** -- yt-dlp version pinned to a pre-2023-07-06 version -- cookies.txt stored in a shared directory (e.g., `/data/cookies.txt` instead of per-session paths) -- cookies.txt files not cleaned up after job completion - -**Phase to address:** -Cookie auth feature (Phase 2 or whenever cookies.txt upload is implemented). Pin the version constraint immediately in Phase 1 setup. - ---- - -### Pitfall 7: SQLite Write Contention Without WAL Mode - -**What goes wrong:** -Multiple concurrent download workers write job status updates (progress %, `downloaded_bytes`, status transitions) to SQLite through aiosqlite. Without WAL mode, SQLite uses a database-level exclusive lock for every write: writer 1 locks, writers 2–N receive `SQLITE_BUSY` and fail (or retry until timeout). Under 3+ simultaneous downloads, status updates are dropped, progress bars freeze, and failed retries surface as 500 errors. - -**Why it happens:** -The default SQLite journal mode (`DELETE`) serializes all writers. aiosqlite runs all operations in a background thread, but the locking is at the database layer, not the Python layer. Developers test with one download at a time and never see contention. - -**How to avoid:** -Enable WAL mode at application startup before any writes: - -```python -async def setup_db(conn): - await conn.execute("PRAGMA journal_mode=WAL") - await conn.execute("PRAGMA synchronous=NORMAL") - await conn.execute("PRAGMA busy_timeout=5000") - await conn.commit() -``` - -`busy_timeout=5000` gives waiting writers up to 5 seconds to retry before failing, absorbing brief contention spikes. WAL allows concurrent readers alongside a single writer, which is exactly the access pattern for a download queue. - -**Warning signs:** -- `sqlite3.OperationalError: database is locked` in logs under concurrent downloads -- Progress bars stall on multiple simultaneous jobs but work fine one at a time -- aiosqlite 0.20.0+ connection thread behavior change causing hangs (ensure connections are properly closed with `async with`) - -**Phase to address:** -Core database setup (Phase 1). Set WAL mode in the database initialization function before any other schema work. - ---- - -## Technical Debt Patterns - -| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable | -|----------|-------------------|----------------|-----------------| -| Single shared aiosqlite connection | Simpler code | Write serialization; connection-level lock defeats WAL concurrency | Never — use a connection pool or per-request connections | -| Hardcoded yt-dlp version (`yt-dlp==2024.x.x`) | Reproducibility | Site extractors break as YouTube/Vimeo update APIs; users report "can't download X" | Acceptable for initial release; add update strategy in v1.1 | -| Storing cookies.txt in a shared `/data/cookies/` directory | Simpler path management | Session A can access session B's cookies if path logic bugs; CVE-2023-35934 surface increases | Never — always per-session isolation | -| Running yt-dlp in the FastAPI process thread pool | No IPC complexity | One hanging download blocks a thread pool slot; OOM in one download can take down the whole process | Acceptable for v1.0 at self-hosted scale; document limit | -| Not implementing `Last-Event-ID` replay at launch | Simpler SSE handler | Every reconnect shows stale/blank UI; impossible to add replay cleanly without event log | Acceptable only if SSE is designed with event IDs from day one so replay can be added later without schema migration | -| `except Exception: pass` in SSE generators | Prevents crashes | Swallows `CancelledError`, creating zombie connections | Never | -| No busy_timeout on SQLite | Fewer config lines | Silent dropped writes under concurrent downloads | Never — always set busy_timeout | - ---- - -## Integration Gotchas - -| Integration | Common Mistake | Correct Approach | -|-------------|----------------|------------------| -| yt-dlp + asyncio | `await loop.run_in_executor(None, ydl.download, [url])` — blocks on `ydl` shared instance | Create `YoutubeDL` inside the worker function; pass only plain data (job_id, url, opts dict) across thread boundary | -| yt-dlp progress hook + event loop | `asyncio.Queue.put_nowait(data)` directly in hook | `loop.call_soon_threadsafe(queue.put_nowait, data)` — capture loop reference before entering executor | -| yt-dlp + ProcessPoolExecutor | Pass `YoutubeDL` instance to process pool | `YoutubeDL` is not picklable (contains file handles); use `ThreadPoolExecutor` only, or create instance inside worker | -| yt-dlp info extraction + download | Call `extract_info` and `download` in same executor call | Fine for ThreadPoolExecutor; `sanitize_info()` required if result crosses process boundary | -| sse-starlette + cleanup | `except Exception as e: cleanup(); pass` | `except asyncio.CancelledError: cleanup(); raise` — never swallow CancelledError | -| aiosqlite 0.20.0+ | `connection.daemon = True` (no longer a thread) | Use `async with aiosqlite.connect()` context manager; verify connection lifecycle in migration from older versions | -| cookies.txt + yt-dlp | Global cookies file path in `YDL_OPTS` shared across requests | Per-session path: `opts["cookiefile"] = f"data/sessions/{session_id}/cookies.txt"` | -| APScheduler + FastAPI lifespan | Starting scheduler outside `@asynccontextmanager lifespan` | Initialize and start scheduler inside the lifespan context manager to ensure clean shutdown | -| Vue 3 EventSource + HTTP/1.1 | Multiple browser tabs each open SSE connection | Serve over HTTP/2 (nginx/caddy in front of uvicorn) to lift 6-connection-per-domain limit | - ---- - -## Performance Traps - -| Trap | Symptoms | Prevention | When It Breaks | -|------|----------|------------|----------------| -| Progress hook writing to DB on every hook call | DB write rate exceeds 10/sec per download; downloads slow down | Throttle DB writes: update DB only when `downloaded_bytes` changes by >1MB or status changes | 3+ simultaneous downloads with fast connections | -| SSE endpoint holding open connection per download per session | Memory grows linearly with active sessions × downloads | One SSE connection per session (multiplexed events), not one per job | 10+ concurrent sessions | -| yt-dlp `extract_info` for URL auto-detection on every keystroke | Rapid URL paste triggers multiple concurrent `extract_info` calls; thread pool saturates | Debounce URL input (500ms) before triggering extraction; cancel in-flight extraction on new input | Immediately, if users paste multi-word text before settling on a URL | -| Docker COPY of entire project directory before pip install | Every code change invalidates pip cache layer | Order Dockerfile: copy `requirements.txt` first → `pip install` → copy app code | Every build during active development | -| aiosqlite without connection pool | Each request opens/closes its own connection; overhead accumulates | Use a single long-lived connection with WAL mode, or `aiosqlitepool` for high throughput | 50+ req/sec (well above self-hosted target, but good practice) | -| Purge scanning entire jobs table without index | Admin-triggered purge takes seconds to complete, blocks event loop if not offloaded | Index `(session_id, status, completed_at)` from the start | 10,000+ job rows | - ---- - -## Security Mistakes - -| Mistake | Risk | Prevention | -|---------|------|------------| -| Cookies.txt stored beyond job lifetime | User's site credentials persist on disk; accessible if container is compromised or volume is shared | Delete on job completion; delete on session purge; include in purge scope always | -| Admin password transmitted without HTTPS | Credentials intercepted on network | Enforce HTTPS in Docker deployment docs; add `SECURE_COOKIES=true` check in startup that warns loudly if running over HTTP | -| Session cookie without `HttpOnly` + `SameSite=Lax` | Cookie accessible via XSS; CSRF possible against download endpoints | Set `response.set_cookie("mrip_session", ..., httponly=True, samesite="lax", secure=False)` (secure=True in prod) | -| Session ID that doesn't rotate after login/admin-auth | Session fixation — attacker sets a known session ID before user authenticates | Regenerate session ID on any privilege change (session creation, admin login) | -| Admin credentials stored in plaintext in `config.yaml` | Credential leak if config volume is readable | Store bcrypt hash of admin password, not plaintext; generate a random default on first boot with forced change prompt | -| yt-dlp version < 2023-07-06 | CVE-2023-35934: cookie leak via redirect | Pin `yt-dlp>=2023.07.06` in `requirements.txt`; verify in Docker health check | -| No rate limiting on download submission | Unauthenticated user floods server with download jobs | Session-scoped queue depth limit (e.g., max 5 active jobs per session); configurable by operator | -| Shareable file URLs that expose internal paths | Directory traversal if filename is user-controlled | Serve files via a controlled endpoint (`/api/files/{job_id}/{filename}`) that resolves to an absolute path; never expose filesystem paths | -| Unsupported URL log with `report_full_url: true` default | Full URLs containing tokens/keys logged and downloadable | Default `report_full_url: false`; document clearly in config reference | - ---- - -## UX Pitfalls - -| Pitfall | User Impact | Better Approach | -|---------|-------------|-----------------| -| "Download failed" with raw yt-dlp error message | Non-technical users see Python tracebacks or opaque errors | Map common yt-dlp errors to human-readable messages: "This site requires login — upload a cookies.txt file" | -| Progress bar resets to 0% on SSE reconnect | User thinks download restarted; anxiety and confusion | Restore last known progress from DB on SSE reconnect; show "Reconnecting..." state briefly | -| Session expiry with no warning | User returns after 24h to find all downloads gone | Show session TTL countdown in UI; warn at 1h remaining; extend TTL on activity | -| Format picker with raw yt-dlp format strings | "bestvideo+bestaudio/best" meaningless to non-technical users | Translate to "Best quality (auto)", "1080p MP4", "Audio only (MP3)"; show file size estimate | -| Playlist shows all items but provides no bulk action | User has to click "start" 40 times for a 40-item playlist | Bulk start at playlist level is required, not optional; implement before any UX testing | -| No feedback when URL auto-detection starts | User pastes URL, nothing visible happens for 2-3 seconds | Show spinner/skeleton immediately on valid URL detection; don't wait for `extract_info` to complete | -| Theme picker that resets on page reload | Users re-select theme every visit | Persist to `localStorage` on selection; read on mount before first render to avoid flash | - ---- - -## "Looks Done But Isn't" Checklist - -- [ ] **Download engine:** Progress hook fires and updates DB — verify that it also correctly handles `total_bytes: None` (subtitle downloads, live streams) without `TypeError` -- [ ] **SSE streaming:** Events deliver in real time on initial connection — verify they also replay correctly after a client disconnect and reconnect using `Last-Event-ID` -- [ ] **Session cookie:** Cookie is set on first visit — verify it has `HttpOnly`, `SameSite=Lax`, and the correct domain/path; verify it is NOT `Secure` in local dev (blocks HTTP) but IS `Secure` in prod -- [ ] **Cookies.txt upload:** File is accepted and passed to yt-dlp — verify the file is deleted after the job completes and is not accessible via any API endpoint -- [ ] **Purge job:** Old jobs are deleted — verify the query explicitly filters `status IN ('completed', 'failed', 'cancelled')` and does not touch `status = 'downloading'` -- [ ] **Admin auth:** Login form accepts correct credentials — verify incorrect credentials return 401 with a constant-time comparison (no timing side channel); verify default credentials force a change prompt -- [ ] **Docker image:** Image builds and runs — verify multi-platform: `docker buildx build --platform linux/amd64,linux/arm64` succeeds before tagging v1.0 -- [ ] **WAL mode:** SQLite is used — verify `PRAGMA journal_mode` returns `wal` at startup in health check or startup log -- [ ] **yt-dlp version:** Library is installed — verify `yt-dlp.__version__` in `/api/health` response and confirm it is >= 2023.07.06 -- [ ] **SSE connection limit:** SSE works in one tab — verify in browser devtools that multiple tabs don't hit HTTP/1.1 6-connection limit (use HTTP/2 or test connection multiplexing) - ---- - -## Recovery Strategies - -| Pitfall | Recovery Cost | Recovery Steps | -|---------|---------------|----------------| -| YoutubeDL instance sharing discovered late | MEDIUM | Audit all `YoutubeDL` instantiation sites; refactor to per-job pattern; existing jobs in-flight are safe (no state corruption once they complete) | -| CancelledError swallowing causing connection leak | LOW | Find `except Exception` blocks in SSE generators; add explicit `except asyncio.CancelledError: raise`; restart server to clear zombie connections | -| Purge bug deleted active download files | LOW | Restore file from backup if available; re-queue job; add status guard to purge query and write regression test | -| cookies.txt not being deleted (security incident) | HIGH | Audit `data/sessions/` directory for leftover cookie files; purge all; rotate any credentials whose cookies were uploaded; add deletion to job completion hook | -| SQLite locked under concurrent downloads | LOW | Enable WAL mode and `busy_timeout`; no data loss if writes are retried; restart not required | -| Docker image too large (>1GB) for arm64 users | MEDIUM | Add `.dockerignore` to exclude `node_modules`, `__pycache__`, `.git`; use multi-stage build with slim Python base; use `wader/static-ffmpeg` for static ffmpeg binary | -| yt-dlp extractor broken by upstream site change | LOW-MEDIUM | Update yt-dlp pin in `requirements.txt` and rebuild image; CI smoke test catches this before release; document manual update procedure in README | - ---- - -## Pitfall-to-Phase Mapping - -| Pitfall | Prevention Phase | Verification | -|---------|------------------|--------------| -| YoutubeDL instance not thread-safe | Phase 1: Core download engine | Test 3 simultaneous downloads; verify no cross-job progress corruption | -| Progress hook not asyncio-safe | Phase 1: Core download engine | Verify SSE receives progress while yt-dlp runs in executor thread | -| SQLite contention without WAL | Phase 1: Database setup | `PRAGMA journal_mode` returns `wal` in startup; no `SQLITE_BUSY` errors under 5 concurrent downloads | -| SSE CancelledError swallowing | Phase 2: SSE streaming | Kill a client mid-stream; verify server task count does not grow over 30 minutes | -| SSE reconnect storm / no replay | Phase 2: SSE streaming | Disconnect and reconnect; verify progress state is restored within 1 SSE cycle | -| cookies.txt leakage | Phase 2: Cookie auth feature | Verify per-session isolation paths; verify file is deleted on job completion | -| Purge deletes active downloads | Phase 3: Purge/session management | Unit test: start slow download, trigger purge, verify file untouched | -| Admin auth security gaps | Phase 3: Admin auth | Verify HttpOnly+SameSite; constant-time password comparison; default password forced change | -| Docker image bloat | Phase 4: Docker distribution | Measure image size post-build: target < 400MB compressed for amd64 | -| yt-dlp version pinning risk | Phase 1: setup + ongoing | `yt-dlp>=2023.07.06` in requirements; health endpoint reports version; CI smoke-test downloads from at least 2 sites | - ---- - -## Sources - -- [yt-dlp issue #9487: asyncio + multiprocessing / YoutubeDL not picklable](https://github.com/yt-dlp/yt-dlp/issues/9487) -- [yt-dlp issue #11022: Concurrent URL downloads not supported natively](https://github.com/yt-dlp/yt-dlp/issues/11022) -- [yt-dlp issue #5957: Progress hooks + writesubtitles / None type error + asyncio incompatibility](https://github.com/yt-dlp/yt-dlp/issues/5957) -- [yt-dlp Security Advisory GHSA-v8mc-9377-rwjj: Cookie leak via redirect (CVE-2023-35934)](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj) -- [sse-starlette: Client Disconnection Detection — CancelledError must be re-raised](https://deepwiki.com/sysid/sse-starlette/3.5-client-disconnection-detection) -- [MDN: Using server-sent events — reconnect and Last-Event-ID behavior](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) -- [SSE production pitfalls: proxy buffering, reconnect, connection limits](https://dev.to/miketalbot/server-sent-events-are-still-not-production-ready-after-a-decade-a-lesson-for-me-a-warning-for-you-2gie) -- [Concurrency challenges in SQLite — write contention and WAL mode](https://www.slingacademy.com/article/concurrency-challenges-in-sqlite-and-how-to-overcome-them/) -- [aiosqlite 0.22.0 behavior change: connection is no longer a thread](https://github.com/sqlalchemy/sqlalchemy/issues/13039) -- [FastAPI SSE disconnect detection discussion](https://github.com/fastapi/fastapi/discussions/9398) -- [Browser connection limits for SSE: 6 per domain on HTTP/1.1](https://www.javascriptroom.com/blog/server-sent-events-and-browser-limits/) -- [wader/static-ffmpeg: multi-arch static ffmpeg binaries for Docker](https://github.com/wader/static-ffmpeg) - ---- -*Pitfalls research for: yt-dlp web frontend (media.rip v1.0)* -*Researched: 2026-03-17* diff --git a/.planning/research/STACK.md b/.planning/research/STACK.md deleted file mode 100644 index a4d8ee8..0000000 --- a/.planning/research/STACK.md +++ /dev/null @@ -1,396 +0,0 @@ -# Stack Research - -**Domain:** Self-hosted yt-dlp web frontend (media downloader) -**Researched:** 2026-03-17 -**Confidence:** HIGH — all versions verified against PyPI and npm as of research date - ---- - -## Recommended Stack - -### Core Technologies - -| Technology | Version | Purpose | Why Recommended | -|------------|---------|---------|-----------------| -| Python | 3.12 | Backend runtime | Pinned in Dockerfile; `3.12-slim` is the smallest viable image. Avoids 3.13's passlib incompatibility. yt-dlp requires >=3.9. | -| FastAPI | 0.135.1 | HTTP API + SSE + middleware | Native SSE support added in 0.135.0 (EventSourceResponse). Async-first design matches the run_in_executor download pattern. HTTPBasic/HTTPBearer auth built in. | -| uvicorn | 0.42.0 | ASGI server | Standard FastAPI server. Use `uvicorn[standard]` for uvloop and httptools for production throughput. | -| yt-dlp | 2026.3.17 | Download engine | Used as a library (`import yt_dlp`), not subprocess. Gives synchronous progress hooks, structured error capture, and no shell-injection surface. | -| aiosqlite | 0.22.1 | Async SQLite | asyncio bridge over stdlib sqlite3. Single-file DB, zero external deps, sufficient for this concurrency model (small ThreadPoolExecutor). | -| APScheduler | 3.11.2 | Cron jobs (purge, session expiry) | 3.x is stable. 4.x is still alpha (4.0.0a6). Use `AsyncIOScheduler` from APScheduler 3.x — runs on FastAPI's event loop, started/stopped in the lifespan context manager. | -| pydantic | 2.12.5 | Data models and validation | FastAPI 0.135.x requires Pydantic v2. All request/response schemas and config validation. | -| pydantic-settings | 2.13.1 | Config loading from YAML + env | Install as `pydantic-settings[yaml]` for native YAML source support. Handles `MEDIARIP__SECTION__KEY` env var override pattern natively with `env_nested_delimiter='__'`. | -| sse-starlette | 3.3.3 | SSE EventSource response | Production-stable. Provides `EventSourceResponse`, handles client disconnect detection, cooperative shutdown, and multiple concurrent streams. Required even though FastAPI 0.135 has native SSE — sse-starlette's disconnect handling is more reliable for long-lived connections. | - -### Supporting Libraries - -| Library | Version | Purpose | When to Use | -|---------|---------|---------|-------------| -| python-multipart | 0.0.22 | Multipart form + file upload | Required for `UploadFile` (cookies.txt upload). FastAPI raises `RuntimeError` without it if any endpoint uses file/form data. | -| bcrypt | 5.0.0 | Password hashing for admin credentials | Direct bcrypt, no passlib wrapper. `bcrypt.hashpw()` / `bcrypt.checkpw()`. Avoids passlib's Python 3.12+ deprecation warnings and Python 3.13 breakage. | -| PyYAML | 6.0.x | YAML parsing for config.yaml | Used indirectly by `pydantic-settings[yaml]`. Pinning to 6.0.x avoids the arbitrary-code-execution issue in 5.x. | -| httpx | 0.28.1 | Async HTTP client for tests | Used with `ASGITransport` for FastAPI integration tests. Not needed at runtime. | -| pytest | 9.0.2 | Backend test runner | Requires Python >=3.10. Use with `anyio` marker for async tests. | -| anyio | bundled with FastAPI | Async test infrastructure | FastAPI uses anyio internally. `@pytest.mark.anyio` with `anyio_backend = "asyncio"` fixture is the correct pattern for async test functions. | -| vue | 3.5.30 | Frontend framework | Latest stable. 3.6.0 is in beta (Vapor mode) — avoid until stable. Composition API + `