From 664eea99778ddb253eaf073bd0db71a06a225695 Mon Sep 17 00:00:00 2001 From: xpltd_admin Date: Fri, 3 Apr 2026 23:04:15 -0600 Subject: [PATCH] Create Architecture wiki page for media-rip --- Architecture.md | 174 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 Architecture.md diff --git a/Architecture.md b/Architecture.md new file mode 100644 index 0000000..d1f5559 --- /dev/null +++ b/Architecture.md @@ -0,0 +1,174 @@ +# Architecture + +| Meta | Value | +|------|-------| +| **Repo** | `xpltdco/media-rip` | +| **Page** | `Architecture` | +| **Audience** | developers, agents | +| **Last Updated** | 2026-04-04 | +| **Status** | current | + +## System Overview + +media-rip is a single-container application with a FastAPI backend and Vue 3 SPA. Downloads run in a thread pool (yt-dlp is synchronous), and real-time progress is delivered to the browser via Server-Sent Events through a thread-safe pub/sub broker. + +```mermaid +graph TB + Browser["Browser (Vue 3 SPA)"] + FastAPI["FastAPI Server :8000"] + Session["Session Middleware"] + Routes["API Routes"] + SSE["SSE /api/events"] + Broker["SSE Broker"] + DLService["Download Service"] + ThreadPool["ThreadPoolExecutor"] + YtDlp["yt-dlp (per-job)"] + DB["SQLite (aiosqlite)"] + Files["/downloads"] + Scheduler["APScheduler"] + Purge["Purge Service"] + + Browser -->|"HTTP"| FastAPI + Browser -->|"EventSource"| SSE + FastAPI --> Session --> Routes + Routes --> DLService + DLService --> ThreadPool + ThreadPool --> YtDlp --> Files + YtDlp -->|"progress hook"| Broker + Broker -->|"push"| SSE --> Browser + Routes --> DB + DLService --> DB + Scheduler -->|"cron"| Purge --> DB + Purge --> Files +``` + +## Tech Stack + +| Layer | Technology | Purpose | +|-------|-----------|---------| +| HTTP Server | FastAPI + Uvicorn | Async ASGI web framework | +| Frontend | Vue 3 + Vite + Pinia | SPA with reactive state management | +| Database | SQLite via aiosqlite | Async embedded database | +| Downloader | yt-dlp + ffmpeg | Media extraction and post-processing | +| Real-time | SSE (sse-starlette) | Server-pushed progress events | +| Scheduler | APScheduler | Cron-based purge automation | +| Auth | bcrypt | Admin password hashing | +| Config | Pydantic Settings + YAML | Multi-source configuration | + +## Directory Structure + +``` +media-rip/ +├── backend/ +│ ├── start.py # Entrypoint (launches uvicorn) +│ ├── app/ +│ │ ├── main.py # FastAPI app + lifespan (init/shutdown) +│ │ ├── dependencies.py # DI: get_session_id, require_admin +│ │ ├── core/ +│ │ │ ├── config.py # Pydantic Settings (env + YAML + defaults) +│ │ │ ├── database.py # SQLite async + WAL/DELETE auto-detection +│ │ │ └── sse_broker.py # Thread-safe pub/sub for SSE +│ │ ├── middleware/ +│ │ │ └── session.py # Cookie session (isolated/shared/open) +│ │ ├── models/ +│ │ │ ├── job.py # Job, JobStatus, FormatInfo, ProgressEvent +│ │ │ └── session.py # Session model +│ │ ├── routers/ +│ │ │ ├── downloads.py # POST/GET/DELETE jobs + URL info +│ │ │ ├── formats.py # GET available formats +│ │ │ ├── sse.py # GET /api/events (EventSource) +│ │ │ ├── admin.py # Admin panel (auth, settings, purge) +│ │ │ ├── cookies.py # Upload/delete cookies.txt +│ │ │ ├── health.py # GET /api/health +│ │ │ ├── system.py # GET /api/config/public +│ │ │ └── themes.py # Custom theme serving +│ │ └── services/ +│ │ ├── download.py # yt-dlp wrapper + ThreadPoolExecutor +│ │ ├── output_template.py # Source-specific path templates +│ │ ├── purge.py # Scheduled cleanup +│ │ ├── settings.py # Admin settings persistence +│ │ └── theme_loader.py # Custom theme scanner +│ ├── tests/ # 14 pytest test files +│ └── requirements.txt # Pinned dependencies +├── frontend/ +│ ├── src/ +│ │ ├── main.ts # Vue app + Pinia + Router +│ │ ├── api/client.ts # API client (fetch wrapper) +│ │ ├── stores/ # Pinia: downloads, config, admin, theme +│ │ ├── composables/useSSE.ts # EventSource lifecycle +│ │ ├── components/ # 11 Vue SFCs +│ │ └── themes/ # 9 built-in theme CSS files +│ ├── package.json +│ └── vite.config.ts +├── Dockerfile # 3-stage build (Node → Python → runtime) +├── docker-compose.yml # Standard deployment +├── docker-compose.example.yml # Secure deployment with Caddy +├── Caddyfile # Auto-TLS reverse proxy +└── .env.example # Configuration template +``` + +## Key Design Decisions + +### Single-container architecture +Unlike multi-service apps, media-rip runs everything in one Docker container. FastAPI serves both the API and the pre-built Vue SPA. SQLite provides persistence without needing PostgreSQL. This keeps deployment dead simple — one container, two volume mounts. + +### Thread pool for yt-dlp +yt-dlp is synchronous and has mutable internal state that isn't thread-safe to share. Each download gets its own yt-dlp instance running in a ThreadPoolExecutor thread. Progress hooks marshal events back to the async context via `loop.call_soon_threadsafe()`. + +### SSE over WebSocket +Server-Sent Events were chosen over WebSocket because progress is unidirectional (server → client). SSE is simpler (plain HTTP), auto-reconnects, and works through proxies without special configuration. The client reconnects with exponential backoff (1s → 30s max). + +### Network filesystem detection +The database module reads `/proc/mounts` to detect if the data directory is on a network filesystem (CIFS, NFS, SMB). If so, it uses SQLite DELETE journal mode instead of WAL (which requires POSIX shared memory). This prevents silent corruption on NAS mounts. + +### Session isolation modes +Three modes support different use cases: **isolated** (multi-user, each browser gets its own queue), **shared** (family/team, everyone sees everything), **open** (no cookies, single session — for kiosks). + +### Multi-source config +Configuration follows a priority chain: environment variables > YAML file > defaults. This means Docker Compose env vars always win, but you can use a `config.yaml` mount for complex setups. Admin-panel changes persist to SQLite and apply on restart. + +## Request Lifecycle + +### Download Request +``` +Browser → POST /api/downloads {url, format_id} + → Session middleware (resolve/create session) + → Validate URL, check API key (if non-browser) + → Create job row (status=queued) in SQLite + → Submit to ThreadPoolExecutor + → Return job object immediately + +Worker thread: + → Create fresh YoutubeDL instance + → Register progress_hook + → Call yt-dlp.download([url]) + → Progress hook fires on every chunk: + → Update DB every 0.5% change + → Publish to SSE broker every 1% change + → On complete: update status=completed, set filename/filesize + → On error: update status=failed, log to error_log table +``` + +### SSE Connection +``` +Browser → GET /api/events (EventSource) + → Session middleware (resolve session) + → SSE broker subscribes a new asyncio.Queue for this session + → Send "init" event with all non-terminal jobs (replay on reconnect) + → Infinite loop: await queue.get() → yield event + → Keepalive: ping every 15s of inactivity + → On disconnect: unsubscribe queue +``` + +## Startup Sequence + +1. Load config (YAML → env → defaults) +2. Hash admin password with bcrypt, clear plaintext from memory +3. Initialize SQLite (auto-detect WAL vs DELETE mode) +4. Recover zombie jobs (queued/downloading → failed) +5. Create SSE broker +6. Create download service (ThreadPoolExecutor) +7. Start purge scheduler (APScheduler cron) +8. Mount middleware (session, security headers) +9. Mount routers + static file serving +10. Start Uvicorn \ No newline at end of file