From 3a9824a2f3abcf06945e9a812c744cbdefc2a9a6 Mon Sep 17 00:00:00 2001 From: xpltd_admin Date: Fri, 3 Apr 2026 22:39:04 -0600 Subject: [PATCH] Create Architecture wiki page with system diagrams and design decisions --- Architecture.md | 232 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 Architecture.md diff --git a/Architecture.md b/Architecture.md new file mode 100644 index 0000000..e481bde --- /dev/null +++ b/Architecture.md @@ -0,0 +1,232 @@ +# Architecture + +| Meta | Value | +|------|-------| +| **Repo** | `xpltdco/tubearr` | +| **Page** | `Architecture` | +| **Audience** | developers, agents, newcomers | +| **Last Updated** | 2026-04-04 | +| **Status** | current | + +## System Overview + +Tubearr is a single-process Node.js application with an embedded SQLite database. The backend (Fastify) serves both the REST API and the React SPA. Downloads are performed by spawning yt-dlp subprocesses. Real-time progress is streamed to the browser via WebSocket. + +```mermaid +graph TB + Browser["Browser (React SPA)"] + Fastify["Fastify Server"] + Auth["Auth Middleware"] + Routes["API Routes"] + WS["WebSocket /ws"] + Scheduler["Scheduler Service"] + Queue["Queue Service"] + Download["Download Service"] + EventBus["Event Bus"] + YtDlp["yt-dlp subprocess"] + DB["SQLite (LibSQL)"] + FileSystem["Media Files (/media)"] + Notify["Notification Service"] + Discord["Discord Webhook"] + + Browser -->|HTTP/WS| Fastify + Fastify --> Auth --> Routes + Fastify --> WS + Routes --> Queue + Routes --> Scheduler + Scheduler -->|"cron jobs"| Queue + Queue -->|"process"| Download + Download --> YtDlp + YtDlp --> FileSystem + Download --> DB + Queue --> Notify --> Discord + EventBus -->|"progress"| WS -->|"push"| Browser + Download -->|"emit"| EventBus + Routes --> DB + Scheduler --> DB +``` + +## Tech Stack + +| Layer | Technology | Purpose | +|-------|-----------|---------| +| HTTP Server | Fastify 5.2.1 | High-performance, plugin-based web framework | +| Frontend | React 19 + Vite | SPA with HMR in development | +| Router | React Router 7 | Client-side page navigation | +| API Client | TanStack Query 5 | Data fetching, caching, background refetch | +| ORM | Drizzle ORM 0.38 | Type-safe SQL queries and schema management | +| Database | SQLite via LibSQL | Embedded database with WAL mode | +| Migrations | Drizzle Kit | Schema change management | +| Downloader | yt-dlp | Media downloading from 1000+ platforms | +| Transcoding | ffmpeg | Post-processing, format conversion | +| Scheduler | croner | Cron-based channel monitoring | +| Icons | Lucide React | UI icon library | +| Testing | Vitest | Unit and integration testing | +| TypeScript | 5.7.3 | Type safety across full stack | + +## Directory Structure + +``` +tubearr/ +├── src/ +│ ├── index.ts # Application entry point & startup orchestration +│ ├── config/ +│ │ └── index.ts # AppConfig — all env vars parsed here +│ ├── db/ +│ │ ├── index.ts # Database init (WAL mode, foreign keys) +│ │ ├── migrate.ts # Migration runner +│ │ ├── schema/ # Drizzle table definitions +│ │ │ ├── channels.ts # Monitored channels +│ │ │ ├── content.ts # Content items + format profiles +│ │ │ ├── queue.ts # Download queue +│ │ │ ├── history.ts # Activity log +│ │ │ ├── notifications.ts # Notification settings +│ │ │ ├── platform-settings.ts # Per-platform defaults +│ │ │ ├── playlists.ts # Playlists + junction table +│ │ │ └── system.ts # Key-value config store +│ │ └── repositories/ # Data access layer (one per table) +│ ├── server/ +│ │ ├── index.ts # Fastify builder, plugin registration +│ │ ├── middleware/ +│ │ │ ├── auth.ts # API key + same-origin auth +│ │ │ └── error-handler.ts # Centralized error responses +│ │ └── routes/ # One file per API resource +│ │ ├── channel.ts # Channel CRUD + scan +│ │ ├── content.ts # Content listing + filters +│ │ ├── download.ts # Enqueue downloads +│ │ ├── queue.ts # Queue management +│ │ ├── format-profile.ts # Quality presets +│ │ ├── notification.ts # Notification config +│ │ ├── platform-settings.ts # Platform defaults +│ │ ├── history.ts # Activity log +│ │ ├── health.ts # GET /ping +│ │ ├── scan.ts # Manual channel scan +│ │ ├── collect.ts # Back-catalog import +│ │ ├── system.ts # API key, settings, yt-dlp +│ │ ├── playlist.ts # Playlist data +│ │ └── websocket.ts # WebSocket progress stream +│ ├── services/ # Business logic layer +│ │ ├── scheduler.ts # Cron-based channel monitoring +│ │ ├── queue.ts # Download queue + concurrency control +│ │ ├── download.ts # Download orchestration (yt-dlp) +│ │ ├── notification.ts # Notification dispatch +│ │ ├── event-bus.ts # Pub/sub for real-time events +│ │ ├── progress-parser.ts # Parse yt-dlp JSON progress +│ │ ├── file-organizer.ts # Organize files by platform/channel +│ │ ├── quality-analyzer.ts # Post-download codec/resolution check +│ │ ├── rate-limiter.ts # Per-platform request throttling +│ │ ├── cookie-manager.ts # Platform cookie persistence +│ │ ├── health.ts # System health diagnostics +│ │ └── back-catalog-import.ts # Bulk media import +│ ├── sources/ # Platform adapters +│ │ ├── platform-source.ts # Abstract interface +│ │ ├── youtube.ts # YouTube channel/content fetcher +│ │ ├── soundcloud.ts # SoundCloud artist/track fetcher +│ │ ├── generic.ts # Fallback URL handler +│ │ └── yt-dlp.ts # yt-dlp subprocess wrapper +│ ├── types/ +│ │ └── index.ts # Shared TypeScript types +│ ├── __tests__/ # 40+ Vitest test suites +│ └── frontend/ +│ ├── src/ +│ │ ├── App.tsx # React router root +│ │ ├── main.tsx # DOM mount +│ │ ├── api/ # API client functions +│ │ ├── components/ # Reusable UI components +│ │ ├── contexts/ # React context (auth, etc.) +│ │ ├── hooks/ # Custom hooks (useChannels, etc.) +│ │ ├── pages/ # Page-level components +│ │ ├── styles/ # CSS +│ │ └── utils/ # Frontend utilities +│ ├── index.html # SPA template +│ ├── vite.config.ts # Vite build config +│ └── tsconfig.json # Frontend TS config +├── drizzle/ # Generated migration SQL files +├── Dockerfile # Multi-stage Docker build +├── docker-compose.yml # Compose orchestration +├── package.json # Dependencies + scripts +├── tsconfig.json # Backend TS config +└── vitest.config.ts # Test config +``` + +## Key Design Decisions + +### SQLite over PostgreSQL +Tubearr uses embedded SQLite (via LibSQL) instead of a separate database server. This simplifies deployment — no external DB container needed — and is sufficient for a single-user media management app. WAL mode enables concurrent reads during writes. + +### Single-process architecture +All services (HTTP, WebSocket, scheduler, queue, downloads) run in one Node.js process. This avoids IPC complexity and keeps the deployment footprint small. The tradeoff is that a crash takes down everything, but the Docker restart policy handles recovery. + +### In-memory queue over external message broker +The download queue runs in-process rather than through Redis/RabbitMQ. Queue state is persisted to SQLite so interrupted downloads survive restarts. This is appropriate for the expected throughput (tens of downloads, not thousands). + +### yt-dlp as subprocess +Rather than reimplementing platform-specific download logic, Tubearr delegates to yt-dlp via child_process. This provides support for 1000+ platforms, SponsorBlock, subtitle handling, and format selection without maintaining any of that code. The tradeoff is a Python runtime dependency in the Docker image. + +### Same-origin auth bypass +Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key. + +### Vite dev middleware in Fastify +In development, Vite's middleware is injected into the Fastify server so both backend and frontend run on port 8989 with HMR. In production, the pre-built SPA is served as static files. This avoids CORS issues and simplifies the dev experience. + +## Service Dependencies + +Tubearr has **no external service dependencies** beyond: +- **yt-dlp** — installed via pip in the Docker image (system dependency) +- **ffmpeg** — installed via apk in the Docker image (system dependency) + +No PostgreSQL, Redis, or other containers required. The Docker image is fully self-contained. + +## Request Lifecycle + +### API Request +``` +Browser/Client → nginx01 (TLS) → Fastify (8989) + → Auth middleware (same-origin check OR API key validation) + → Route handler (validate input, call service/repository) + → Drizzle ORM → SQLite (WAL mode) + → JSON response +``` + +### Download Flow +``` +User clicks "Download" → POST /api/v1/download/:id + → QueueService.enqueue() → queue_items row (pending) + → QueueService.processNext() picks up item + → RateLimiter.acquire(platform) → wait if throttled + → DownloadService.downloadItem() + → Build yt-dlp args from format profile + → Spawn yt-dlp subprocess + → ProgressParser streams JSON → EventBus → WebSocket → Browser + → FileOrganizer.organize() → move to /media/Platform/Channel/ + → QualityAnalyzer.analyze() → inspect codec/resolution + → Update content_items (filePath, fileSize, status=downloaded) + → NotificationService.dispatch(onDownload) +``` + +### Channel Monitoring +``` +SchedulerService starts cron job per channel (e.g., every 6 hours) + → PlatformSource.fetchRecentContent(channel) + → yt-dlp --flat-playlist --dump-single-json + → Parse response → list of content items + → Deduplicate against existing platformContentIds in DB + → Insert new content_items (status=monitored) + → If monitoringMode includes auto-download → enqueue items + → Update lastCheckedAt, lastCheckStatus +``` + +## Startup Sequence + +1. Load `.env` via dotenv +2. Initialize SQLite with WAL pragmas (`journal_mode`, `busy_timeout`, `foreign_keys`) +3. Run Drizzle migrations (idempotent) +4. Seed default format profile and system config +5. Check/update yt-dlp version (production only) +6. Create Vite dev server (development only) +7. Build Fastify server — register plugins, middleware, routes +8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService +9. Register platform sources (YouTube, SoundCloud, Generic) +10. Start Fastify on `TUBEARR_PORT` +11. Recover interrupted queue items → restart processing +12. Start scheduler (monitor enabled channels) \ No newline at end of file