Create Architecture wiki page for media-rip

xpltd_admin 2026-04-03 23:04:15 -06:00
parent 7f897b6e26
commit 664eea9977

174
Architecture.md Normal file

@ -0,0 +1,174 @@
# Architecture
| Meta | Value |
|------|-------|
| **Repo** | `xpltdco/media-rip` |
| **Page** | `Architecture` |
| **Audience** | developers, agents |
| **Last Updated** | 2026-04-04 |
| **Status** | current |
## System Overview
media-rip is a single-container application with a FastAPI backend and Vue 3 SPA. Downloads run in a thread pool (yt-dlp is synchronous), and real-time progress is delivered to the browser via Server-Sent Events through a thread-safe pub/sub broker.
```mermaid
graph TB
Browser["Browser (Vue 3 SPA)"]
FastAPI["FastAPI Server :8000"]
Session["Session Middleware"]
Routes["API Routes"]
SSE["SSE /api/events"]
Broker["SSE Broker"]
DLService["Download Service"]
ThreadPool["ThreadPoolExecutor"]
YtDlp["yt-dlp (per-job)"]
DB["SQLite (aiosqlite)"]
Files["/downloads"]
Scheduler["APScheduler"]
Purge["Purge Service"]
Browser -->|"HTTP"| FastAPI
Browser -->|"EventSource"| SSE
FastAPI --> Session --> Routes
Routes --> DLService
DLService --> ThreadPool
ThreadPool --> YtDlp --> Files
YtDlp -->|"progress hook"| Broker
Broker -->|"push"| SSE --> Browser
Routes --> DB
DLService --> DB
Scheduler -->|"cron"| Purge --> DB
Purge --> Files
```
## Tech Stack
| Layer | Technology | Purpose |
|-------|-----------|---------|
| HTTP Server | FastAPI + Uvicorn | Async ASGI web framework |
| Frontend | Vue 3 + Vite + Pinia | SPA with reactive state management |
| Database | SQLite via aiosqlite | Async embedded database |
| Downloader | yt-dlp + ffmpeg | Media extraction and post-processing |
| Real-time | SSE (sse-starlette) | Server-pushed progress events |
| Scheduler | APScheduler | Cron-based purge automation |
| Auth | bcrypt | Admin password hashing |
| Config | Pydantic Settings + YAML | Multi-source configuration |
## Directory Structure
```
media-rip/
├── backend/
│ ├── start.py # Entrypoint (launches uvicorn)
│ ├── app/
│ │ ├── main.py # FastAPI app + lifespan (init/shutdown)
│ │ ├── dependencies.py # DI: get_session_id, require_admin
│ │ ├── core/
│ │ │ ├── config.py # Pydantic Settings (env + YAML + defaults)
│ │ │ ├── database.py # SQLite async + WAL/DELETE auto-detection
│ │ │ └── sse_broker.py # Thread-safe pub/sub for SSE
│ │ ├── middleware/
│ │ │ └── session.py # Cookie session (isolated/shared/open)
│ │ ├── models/
│ │ │ ├── job.py # Job, JobStatus, FormatInfo, ProgressEvent
│ │ │ └── session.py # Session model
│ │ ├── routers/
│ │ │ ├── downloads.py # POST/GET/DELETE jobs + URL info
│ │ │ ├── formats.py # GET available formats
│ │ │ ├── sse.py # GET /api/events (EventSource)
│ │ │ ├── admin.py # Admin panel (auth, settings, purge)
│ │ │ ├── cookies.py # Upload/delete cookies.txt
│ │ │ ├── health.py # GET /api/health
│ │ │ ├── system.py # GET /api/config/public
│ │ │ └── themes.py # Custom theme serving
│ │ └── services/
│ │ ├── download.py # yt-dlp wrapper + ThreadPoolExecutor
│ │ ├── output_template.py # Source-specific path templates
│ │ ├── purge.py # Scheduled cleanup
│ │ ├── settings.py # Admin settings persistence
│ │ └── theme_loader.py # Custom theme scanner
│ ├── tests/ # 14 pytest test files
│ └── requirements.txt # Pinned dependencies
├── frontend/
│ ├── src/
│ │ ├── main.ts # Vue app + Pinia + Router
│ │ ├── api/client.ts # API client (fetch wrapper)
│ │ ├── stores/ # Pinia: downloads, config, admin, theme
│ │ ├── composables/useSSE.ts # EventSource lifecycle
│ │ ├── components/ # 11 Vue SFCs
│ │ └── themes/ # 9 built-in theme CSS files
│ ├── package.json
│ └── vite.config.ts
├── Dockerfile # 3-stage build (Node → Python → runtime)
├── docker-compose.yml # Standard deployment
├── docker-compose.example.yml # Secure deployment with Caddy
├── Caddyfile # Auto-TLS reverse proxy
└── .env.example # Configuration template
```
## Key Design Decisions
### Single-container architecture
Unlike multi-service apps, media-rip runs everything in one Docker container. FastAPI serves both the API and the pre-built Vue SPA. SQLite provides persistence without needing PostgreSQL. This keeps deployment dead simple — one container, two volume mounts.
### Thread pool for yt-dlp
yt-dlp is synchronous and has mutable internal state that isn't thread-safe to share. Each download gets its own yt-dlp instance running in a ThreadPoolExecutor thread. Progress hooks marshal events back to the async context via `loop.call_soon_threadsafe()`.
### SSE over WebSocket
Server-Sent Events were chosen over WebSocket because progress is unidirectional (server → client). SSE is simpler (plain HTTP), auto-reconnects, and works through proxies without special configuration. The client reconnects with exponential backoff (1s → 30s max).
### Network filesystem detection
The database module reads `/proc/mounts` to detect if the data directory is on a network filesystem (CIFS, NFS, SMB). If so, it uses SQLite DELETE journal mode instead of WAL (which requires POSIX shared memory). This prevents silent corruption on NAS mounts.
### Session isolation modes
Three modes support different use cases: **isolated** (multi-user, each browser gets its own queue), **shared** (family/team, everyone sees everything), **open** (no cookies, single session — for kiosks).
### Multi-source config
Configuration follows a priority chain: environment variables > YAML file > defaults. This means Docker Compose env vars always win, but you can use a `config.yaml` mount for complex setups. Admin-panel changes persist to SQLite and apply on restart.
## Request Lifecycle
### Download Request
```
Browser → POST /api/downloads {url, format_id}
→ Session middleware (resolve/create session)
→ Validate URL, check API key (if non-browser)
→ Create job row (status=queued) in SQLite
→ Submit to ThreadPoolExecutor
→ Return job object immediately
Worker thread:
→ Create fresh YoutubeDL instance
→ Register progress_hook
→ Call yt-dlp.download([url])
→ Progress hook fires on every chunk:
→ Update DB every 0.5% change
→ Publish to SSE broker every 1% change
→ On complete: update status=completed, set filename/filesize
→ On error: update status=failed, log to error_log table
```
### SSE Connection
```
Browser → GET /api/events (EventSource)
→ Session middleware (resolve session)
→ SSE broker subscribes a new asyncio.Queue for this session
→ Send "init" event with all non-terminal jobs (replay on reconnect)
→ Infinite loop: await queue.get() → yield event
→ Keepalive: ping every 15s of inactivity
→ On disconnect: unsubscribe queue
```
## Startup Sequence
1. Load config (YAML → env → defaults)
2. Hash admin password with bcrypt, clear plaintext from memory
3. Initialize SQLite (auto-detect WAL vs DELETE mode)
4. Recover zombie jobs (queued/downloading → failed)
5. Create SSE broker
6. Create download service (ThreadPoolExecutor)
7. Start purge scheduler (APScheduler cron)
8. Mount middleware (session, security headers)
9. Mount routers + static file serving
10. Start Uvicorn