mirror of
https://github.com/xpltdco/media-rip.git
synced 2026-06-02 10:54:29 -06:00
Create Architecture wiki page for media-rip
parent
7f897b6e26
commit
664eea9977
1 changed files with 174 additions and 0 deletions
174
Architecture.md
Normal file
174
Architecture.md
Normal file
|
|
@ -0,0 +1,174 @@
|
||||||
|
# Architecture
|
||||||
|
|
||||||
|
| Meta | Value |
|
||||||
|
|------|-------|
|
||||||
|
| **Repo** | `xpltdco/media-rip` |
|
||||||
|
| **Page** | `Architecture` |
|
||||||
|
| **Audience** | developers, agents |
|
||||||
|
| **Last Updated** | 2026-04-04 |
|
||||||
|
| **Status** | current |
|
||||||
|
|
||||||
|
## System Overview
|
||||||
|
|
||||||
|
media-rip is a single-container application with a FastAPI backend and Vue 3 SPA. Downloads run in a thread pool (yt-dlp is synchronous), and real-time progress is delivered to the browser via Server-Sent Events through a thread-safe pub/sub broker.
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TB
|
||||||
|
Browser["Browser (Vue 3 SPA)"]
|
||||||
|
FastAPI["FastAPI Server :8000"]
|
||||||
|
Session["Session Middleware"]
|
||||||
|
Routes["API Routes"]
|
||||||
|
SSE["SSE /api/events"]
|
||||||
|
Broker["SSE Broker"]
|
||||||
|
DLService["Download Service"]
|
||||||
|
ThreadPool["ThreadPoolExecutor"]
|
||||||
|
YtDlp["yt-dlp (per-job)"]
|
||||||
|
DB["SQLite (aiosqlite)"]
|
||||||
|
Files["/downloads"]
|
||||||
|
Scheduler["APScheduler"]
|
||||||
|
Purge["Purge Service"]
|
||||||
|
|
||||||
|
Browser -->|"HTTP"| FastAPI
|
||||||
|
Browser -->|"EventSource"| SSE
|
||||||
|
FastAPI --> Session --> Routes
|
||||||
|
Routes --> DLService
|
||||||
|
DLService --> ThreadPool
|
||||||
|
ThreadPool --> YtDlp --> Files
|
||||||
|
YtDlp -->|"progress hook"| Broker
|
||||||
|
Broker -->|"push"| SSE --> Browser
|
||||||
|
Routes --> DB
|
||||||
|
DLService --> DB
|
||||||
|
Scheduler -->|"cron"| Purge --> DB
|
||||||
|
Purge --> Files
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tech Stack
|
||||||
|
|
||||||
|
| Layer | Technology | Purpose |
|
||||||
|
|-------|-----------|---------|
|
||||||
|
| HTTP Server | FastAPI + Uvicorn | Async ASGI web framework |
|
||||||
|
| Frontend | Vue 3 + Vite + Pinia | SPA with reactive state management |
|
||||||
|
| Database | SQLite via aiosqlite | Async embedded database |
|
||||||
|
| Downloader | yt-dlp + ffmpeg | Media extraction and post-processing |
|
||||||
|
| Real-time | SSE (sse-starlette) | Server-pushed progress events |
|
||||||
|
| Scheduler | APScheduler | Cron-based purge automation |
|
||||||
|
| Auth | bcrypt | Admin password hashing |
|
||||||
|
| Config | Pydantic Settings + YAML | Multi-source configuration |
|
||||||
|
|
||||||
|
## Directory Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
media-rip/
|
||||||
|
├── backend/
|
||||||
|
│ ├── start.py # Entrypoint (launches uvicorn)
|
||||||
|
│ ├── app/
|
||||||
|
│ │ ├── main.py # FastAPI app + lifespan (init/shutdown)
|
||||||
|
│ │ ├── dependencies.py # DI: get_session_id, require_admin
|
||||||
|
│ │ ├── core/
|
||||||
|
│ │ │ ├── config.py # Pydantic Settings (env + YAML + defaults)
|
||||||
|
│ │ │ ├── database.py # SQLite async + WAL/DELETE auto-detection
|
||||||
|
│ │ │ └── sse_broker.py # Thread-safe pub/sub for SSE
|
||||||
|
│ │ ├── middleware/
|
||||||
|
│ │ │ └── session.py # Cookie session (isolated/shared/open)
|
||||||
|
│ │ ├── models/
|
||||||
|
│ │ │ ├── job.py # Job, JobStatus, FormatInfo, ProgressEvent
|
||||||
|
│ │ │ └── session.py # Session model
|
||||||
|
│ │ ├── routers/
|
||||||
|
│ │ │ ├── downloads.py # POST/GET/DELETE jobs + URL info
|
||||||
|
│ │ │ ├── formats.py # GET available formats
|
||||||
|
│ │ │ ├── sse.py # GET /api/events (EventSource)
|
||||||
|
│ │ │ ├── admin.py # Admin panel (auth, settings, purge)
|
||||||
|
│ │ │ ├── cookies.py # Upload/delete cookies.txt
|
||||||
|
│ │ │ ├── health.py # GET /api/health
|
||||||
|
│ │ │ ├── system.py # GET /api/config/public
|
||||||
|
│ │ │ └── themes.py # Custom theme serving
|
||||||
|
│ │ └── services/
|
||||||
|
│ │ ├── download.py # yt-dlp wrapper + ThreadPoolExecutor
|
||||||
|
│ │ ├── output_template.py # Source-specific path templates
|
||||||
|
│ │ ├── purge.py # Scheduled cleanup
|
||||||
|
│ │ ├── settings.py # Admin settings persistence
|
||||||
|
│ │ └── theme_loader.py # Custom theme scanner
|
||||||
|
│ ├── tests/ # 14 pytest test files
|
||||||
|
│ └── requirements.txt # Pinned dependencies
|
||||||
|
├── frontend/
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── main.ts # Vue app + Pinia + Router
|
||||||
|
│ │ ├── api/client.ts # API client (fetch wrapper)
|
||||||
|
│ │ ├── stores/ # Pinia: downloads, config, admin, theme
|
||||||
|
│ │ ├── composables/useSSE.ts # EventSource lifecycle
|
||||||
|
│ │ ├── components/ # 11 Vue SFCs
|
||||||
|
│ │ └── themes/ # 9 built-in theme CSS files
|
||||||
|
│ ├── package.json
|
||||||
|
│ └── vite.config.ts
|
||||||
|
├── Dockerfile # 3-stage build (Node → Python → runtime)
|
||||||
|
├── docker-compose.yml # Standard deployment
|
||||||
|
├── docker-compose.example.yml # Secure deployment with Caddy
|
||||||
|
├── Caddyfile # Auto-TLS reverse proxy
|
||||||
|
└── .env.example # Configuration template
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Design Decisions
|
||||||
|
|
||||||
|
### Single-container architecture
|
||||||
|
Unlike multi-service apps, media-rip runs everything in one Docker container. FastAPI serves both the API and the pre-built Vue SPA. SQLite provides persistence without needing PostgreSQL. This keeps deployment dead simple — one container, two volume mounts.
|
||||||
|
|
||||||
|
### Thread pool for yt-dlp
|
||||||
|
yt-dlp is synchronous and has mutable internal state that isn't thread-safe to share. Each download gets its own yt-dlp instance running in a ThreadPoolExecutor thread. Progress hooks marshal events back to the async context via `loop.call_soon_threadsafe()`.
|
||||||
|
|
||||||
|
### SSE over WebSocket
|
||||||
|
Server-Sent Events were chosen over WebSocket because progress is unidirectional (server → client). SSE is simpler (plain HTTP), auto-reconnects, and works through proxies without special configuration. The client reconnects with exponential backoff (1s → 30s max).
|
||||||
|
|
||||||
|
### Network filesystem detection
|
||||||
|
The database module reads `/proc/mounts` to detect if the data directory is on a network filesystem (CIFS, NFS, SMB). If so, it uses SQLite DELETE journal mode instead of WAL (which requires POSIX shared memory). This prevents silent corruption on NAS mounts.
|
||||||
|
|
||||||
|
### Session isolation modes
|
||||||
|
Three modes support different use cases: **isolated** (multi-user, each browser gets its own queue), **shared** (family/team, everyone sees everything), **open** (no cookies, single session — for kiosks).
|
||||||
|
|
||||||
|
### Multi-source config
|
||||||
|
Configuration follows a priority chain: environment variables > YAML file > defaults. This means Docker Compose env vars always win, but you can use a `config.yaml` mount for complex setups. Admin-panel changes persist to SQLite and apply on restart.
|
||||||
|
|
||||||
|
## Request Lifecycle
|
||||||
|
|
||||||
|
### Download Request
|
||||||
|
```
|
||||||
|
Browser → POST /api/downloads {url, format_id}
|
||||||
|
→ Session middleware (resolve/create session)
|
||||||
|
→ Validate URL, check API key (if non-browser)
|
||||||
|
→ Create job row (status=queued) in SQLite
|
||||||
|
→ Submit to ThreadPoolExecutor
|
||||||
|
→ Return job object immediately
|
||||||
|
|
||||||
|
Worker thread:
|
||||||
|
→ Create fresh YoutubeDL instance
|
||||||
|
→ Register progress_hook
|
||||||
|
→ Call yt-dlp.download([url])
|
||||||
|
→ Progress hook fires on every chunk:
|
||||||
|
→ Update DB every 0.5% change
|
||||||
|
→ Publish to SSE broker every 1% change
|
||||||
|
→ On complete: update status=completed, set filename/filesize
|
||||||
|
→ On error: update status=failed, log to error_log table
|
||||||
|
```
|
||||||
|
|
||||||
|
### SSE Connection
|
||||||
|
```
|
||||||
|
Browser → GET /api/events (EventSource)
|
||||||
|
→ Session middleware (resolve session)
|
||||||
|
→ SSE broker subscribes a new asyncio.Queue for this session
|
||||||
|
→ Send "init" event with all non-terminal jobs (replay on reconnect)
|
||||||
|
→ Infinite loop: await queue.get() → yield event
|
||||||
|
→ Keepalive: ping every 15s of inactivity
|
||||||
|
→ On disconnect: unsubscribe queue
|
||||||
|
```
|
||||||
|
|
||||||
|
## Startup Sequence
|
||||||
|
|
||||||
|
1. Load config (YAML → env → defaults)
|
||||||
|
2. Hash admin password with bcrypt, clear plaintext from memory
|
||||||
|
3. Initialize SQLite (auto-detect WAL vs DELETE mode)
|
||||||
|
4. Recover zombie jobs (queued/downloading → failed)
|
||||||
|
5. Create SSE broker
|
||||||
|
6. Create download service (ThreadPoolExecutor)
|
||||||
|
7. Start purge scheduler (APScheduler cron)
|
||||||
|
8. Mount middleware (session, security headers)
|
||||||
|
9. Mount routers + static file serving
|
||||||
|
10. Start Uvicorn
|
||||||
Loading…
Add table
Reference in a new issue