Create Architecture wiki page with system diagrams and design decisions

2026-04-03 22:39:04 -06:00 · 2026-04-03 22:39:04 -06:00 · 3a9824a2f3
commit 3a9824a2f3
parent c8a6d90499
1 changed files with 232 additions and 0 deletions
--- a/Architecture.md
+++ b/Architecture.md
@ -0,0 +1,232 @@
+# Architecture
+
+| Meta | Value |
+|------|-------|
+| **Repo** | `xpltdco/tubearr` |
+| **Page** | `Architecture` |
+| **Audience** | developers, agents, newcomers |
+| **Last Updated** | 2026-04-04 |
+| **Status** | current |
+
+## System Overview
+
+Tubearr is a single-process Node.js application with an embedded SQLite database. The backend (Fastify) serves both the REST API and the React SPA. Downloads are performed by spawning yt-dlp subprocesses. Real-time progress is streamed to the browser via WebSocket.
+
+```mermaid
+graph TB
+    Browser["Browser (React SPA)"]
+    Fastify["Fastify Server"]
+    Auth["Auth Middleware"]
+    Routes["API Routes"]
+    WS["WebSocket /ws"]
+    Scheduler["Scheduler Service"]
+    Queue["Queue Service"]
+    Download["Download Service"]
+    EventBus["Event Bus"]
+    YtDlp["yt-dlp subprocess"]
+    DB["SQLite (LibSQL)"]
+    FileSystem["Media Files (/media)"]
+    Notify["Notification Service"]
+    Discord["Discord Webhook"]
+
+    Browser -->|HTTP/WS| Fastify
+    Fastify --> Auth --> Routes
+    Fastify --> WS
+    Routes --> Queue
+    Routes --> Scheduler
+    Scheduler -->|"cron jobs"| Queue
+    Queue -->|"process"| Download
+    Download --> YtDlp
+    YtDlp --> FileSystem
+    Download --> DB
+    Queue --> Notify --> Discord
+    EventBus -->|"progress"| WS -->|"push"| Browser
+    Download -->|"emit"| EventBus
+    Routes --> DB
+    Scheduler --> DB
+```
+
+## Tech Stack
+
+| Layer | Technology | Purpose |
+|-------|-----------|---------|
+| HTTP Server | Fastify 5.2.1 | High-performance, plugin-based web framework |
+| Frontend | React 19 + Vite | SPA with HMR in development |
+| Router | React Router 7 | Client-side page navigation |
+| API Client | TanStack Query 5 | Data fetching, caching, background refetch |
+| ORM | Drizzle ORM 0.38 | Type-safe SQL queries and schema management |
+| Database | SQLite via LibSQL | Embedded database with WAL mode |
+| Migrations | Drizzle Kit | Schema change management |
+| Downloader | yt-dlp | Media downloading from 1000+ platforms |
+| Transcoding | ffmpeg | Post-processing, format conversion |
+| Scheduler | croner | Cron-based channel monitoring |
+| Icons | Lucide React | UI icon library |
+| Testing | Vitest | Unit and integration testing |
+| TypeScript | 5.7.3 | Type safety across full stack |
+
+## Directory Structure
+
+```
+tubearr/
+├── src/
+│   ├── index.ts                    # Application entry point & startup orchestration
+│   ├── config/
+│   │   └── index.ts                # AppConfig — all env vars parsed here
+│   ├── db/
+│   │   ├── index.ts                # Database init (WAL mode, foreign keys)
+│   │   ├── migrate.ts              # Migration runner
+│   │   ├── schema/                 # Drizzle table definitions
+│   │   │   ├── channels.ts         # Monitored channels
+│   │   │   ├── content.ts          # Content items + format profiles
+│   │   │   ├── queue.ts            # Download queue
+│   │   │   ├── history.ts          # Activity log
+│   │   │   ├── notifications.ts    # Notification settings
+│   │   │   ├── platform-settings.ts # Per-platform defaults
+│   │   │   ├── playlists.ts        # Playlists + junction table
+│   │   │   └── system.ts           # Key-value config store
+│   │   └── repositories/           # Data access layer (one per table)
+│   ├── server/
+│   │   ├── index.ts                # Fastify builder, plugin registration
+│   │   ├── middleware/
+│   │   │   ├── auth.ts             # API key + same-origin auth
+│   │   │   └── error-handler.ts    # Centralized error responses
+│   │   └── routes/                 # One file per API resource
+│   │       ├── channel.ts          # Channel CRUD + scan
+│   │       ├── content.ts          # Content listing + filters
+│   │       ├── download.ts         # Enqueue downloads
+│   │       ├── queue.ts            # Queue management
+│   │       ├── format-profile.ts   # Quality presets
+│   │       ├── notification.ts     # Notification config
+│   │       ├── platform-settings.ts # Platform defaults
+│   │       ├── history.ts          # Activity log
+│   │       ├── health.ts           # GET /ping
+│   │       ├── scan.ts             # Manual channel scan
+│   │       ├── collect.ts          # Back-catalog import
+│   │       ├── system.ts           # API key, settings, yt-dlp
+│   │       ├── playlist.ts         # Playlist data
+│   │       └── websocket.ts        # WebSocket progress stream
+│   ├── services/                   # Business logic layer
+│   │   ├── scheduler.ts            # Cron-based channel monitoring
+│   │   ├── queue.ts                # Download queue + concurrency control
+│   │   ├── download.ts             # Download orchestration (yt-dlp)
+│   │   ├── notification.ts         # Notification dispatch
+│   │   ├── event-bus.ts            # Pub/sub for real-time events
+│   │   ├── progress-parser.ts      # Parse yt-dlp JSON progress
+│   │   ├── file-organizer.ts       # Organize files by platform/channel
+│   │   ├── quality-analyzer.ts     # Post-download codec/resolution check
+│   │   ├── rate-limiter.ts         # Per-platform request throttling
+│   │   ├── cookie-manager.ts       # Platform cookie persistence
+│   │   ├── health.ts               # System health diagnostics
+│   │   └── back-catalog-import.ts  # Bulk media import
+│   ├── sources/                    # Platform adapters
+│   │   ├── platform-source.ts      # Abstract interface
+│   │   ├── youtube.ts              # YouTube channel/content fetcher
+│   │   ├── soundcloud.ts           # SoundCloud artist/track fetcher
+│   │   ├── generic.ts              # Fallback URL handler
+│   │   └── yt-dlp.ts               # yt-dlp subprocess wrapper
+│   ├── types/
+│   │   └── index.ts                # Shared TypeScript types
+│   ├── __tests__/                  # 40+ Vitest test suites
+│   └── frontend/
+│       ├── src/
+│       │   ├── App.tsx             # React router root
+│       │   ├── main.tsx            # DOM mount
+│       │   ├── api/                # API client functions
+│       │   ├── components/         # Reusable UI components
+│       │   ├── contexts/           # React context (auth, etc.)
+│       │   ├── hooks/              # Custom hooks (useChannels, etc.)
+│       │   ├── pages/              # Page-level components
+│       │   ├── styles/             # CSS
+│       │   └── utils/              # Frontend utilities
+│       ├── index.html              # SPA template
+│       ├── vite.config.ts          # Vite build config
+│       └── tsconfig.json           # Frontend TS config
+├── drizzle/                        # Generated migration SQL files
+├── Dockerfile                      # Multi-stage Docker build
+├── docker-compose.yml              # Compose orchestration
+├── package.json                    # Dependencies + scripts
+├── tsconfig.json                   # Backend TS config
+└── vitest.config.ts                # Test config
+```
+
+## Key Design Decisions
+
+### SQLite over PostgreSQL
+Tubearr uses embedded SQLite (via LibSQL) instead of a separate database server. This simplifies deployment — no external DB container needed — and is sufficient for a single-user media management app. WAL mode enables concurrent reads during writes.
+
+### Single-process architecture
+All services (HTTP, WebSocket, scheduler, queue, downloads) run in one Node.js process. This avoids IPC complexity and keeps the deployment footprint small. The tradeoff is that a crash takes down everything, but the Docker restart policy handles recovery.
+
+### In-memory queue over external message broker
+The download queue runs in-process rather than through Redis/RabbitMQ. Queue state is persisted to SQLite so interrupted downloads survive restarts. This is appropriate for the expected throughput (tens of downloads, not thousands).
+
+### yt-dlp as subprocess
+Rather than reimplementing platform-specific download logic, Tubearr delegates to yt-dlp via child_process. This provides support for 1000+ platforms, SponsorBlock, subtitle handling, and format selection without maintaining any of that code. The tradeoff is a Python runtime dependency in the Docker image.
+
+### Same-origin auth bypass
+Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key.
+
+### Vite dev middleware in Fastify
+In development, Vite's middleware is injected into the Fastify server so both backend and frontend run on port 8989 with HMR. In production, the pre-built SPA is served as static files. This avoids CORS issues and simplifies the dev experience.
+
+## Service Dependencies
+
+Tubearr has **no external service dependencies** beyond:
+- **yt-dlp** — installed via pip in the Docker image (system dependency)
+- **ffmpeg** — installed via apk in the Docker image (system dependency)
+
+No PostgreSQL, Redis, or other containers required. The Docker image is fully self-contained.
+
+## Request Lifecycle
+
+### API Request
+```
+Browser/Client → nginx01 (TLS) → Fastify (8989)
+  → Auth middleware (same-origin check OR API key validation)
+  → Route handler (validate input, call service/repository)
+  → Drizzle ORM → SQLite (WAL mode)
+  → JSON response
+```
+
+### Download Flow
+```
+User clicks "Download" → POST /api/v1/download/:id
+  → QueueService.enqueue() → queue_items row (pending)
+  → QueueService.processNext() picks up item
+  → RateLimiter.acquire(platform) → wait if throttled
+  → DownloadService.downloadItem()
+    → Build yt-dlp args from format profile
+    → Spawn yt-dlp subprocess
+    → ProgressParser streams JSON → EventBus → WebSocket → Browser
+    → FileOrganizer.organize() → move to /media/Platform/Channel/
+    → QualityAnalyzer.analyze() → inspect codec/resolution
+    → Update content_items (filePath, fileSize, status=downloaded)
+  → NotificationService.dispatch(onDownload)
+```
+
+### Channel Monitoring
+```
+SchedulerService starts cron job per channel (e.g., every 6 hours)
+  → PlatformSource.fetchRecentContent(channel)
+    → yt-dlp --flat-playlist --dump-single-json
+    → Parse response → list of content items
+  → Deduplicate against existing platformContentIds in DB
+  → Insert new content_items (status=monitored)
+  → If monitoringMode includes auto-download → enqueue items
+  → Update lastCheckedAt, lastCheckStatus
+```
+
+## Startup Sequence
+
+1. Load `.env` via dotenv
+2. Initialize SQLite with WAL pragmas (`journal_mode`, `busy_timeout`, `foreign_keys`)
+3. Run Drizzle migrations (idempotent)
+4. Seed default format profile and system config
+5. Check/update yt-dlp version (production only)
+6. Create Vite dev server (development only)
+7. Build Fastify server — register plugins, middleware, routes
+8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService
+9. Register platform sources (YouTube, SoundCloud, Generic)
+10. Start Fastify on `TUBEARR_PORT`
+11. Recover interrupted queue items → restart processing
+12. Start scheduler (monitor enabled channels)