Create Architecture wiki page with system diagrams and design decisions
parent
c8a6d90499
commit
3a9824a2f3
1 changed files with 232 additions and 0 deletions
232
Architecture.md
Normal file
232
Architecture.md
Normal file
|
|
@ -0,0 +1,232 @@
|
||||||
|
# Architecture
|
||||||
|
|
||||||
|
| Meta | Value |
|
||||||
|
|------|-------|
|
||||||
|
| **Repo** | `xpltdco/tubearr` |
|
||||||
|
| **Page** | `Architecture` |
|
||||||
|
| **Audience** | developers, agents, newcomers |
|
||||||
|
| **Last Updated** | 2026-04-04 |
|
||||||
|
| **Status** | current |
|
||||||
|
|
||||||
|
## System Overview
|
||||||
|
|
||||||
|
Tubearr is a single-process Node.js application with an embedded SQLite database. The backend (Fastify) serves both the REST API and the React SPA. Downloads are performed by spawning yt-dlp subprocesses. Real-time progress is streamed to the browser via WebSocket.
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TB
|
||||||
|
Browser["Browser (React SPA)"]
|
||||||
|
Fastify["Fastify Server"]
|
||||||
|
Auth["Auth Middleware"]
|
||||||
|
Routes["API Routes"]
|
||||||
|
WS["WebSocket /ws"]
|
||||||
|
Scheduler["Scheduler Service"]
|
||||||
|
Queue["Queue Service"]
|
||||||
|
Download["Download Service"]
|
||||||
|
EventBus["Event Bus"]
|
||||||
|
YtDlp["yt-dlp subprocess"]
|
||||||
|
DB["SQLite (LibSQL)"]
|
||||||
|
FileSystem["Media Files (/media)"]
|
||||||
|
Notify["Notification Service"]
|
||||||
|
Discord["Discord Webhook"]
|
||||||
|
|
||||||
|
Browser -->|HTTP/WS| Fastify
|
||||||
|
Fastify --> Auth --> Routes
|
||||||
|
Fastify --> WS
|
||||||
|
Routes --> Queue
|
||||||
|
Routes --> Scheduler
|
||||||
|
Scheduler -->|"cron jobs"| Queue
|
||||||
|
Queue -->|"process"| Download
|
||||||
|
Download --> YtDlp
|
||||||
|
YtDlp --> FileSystem
|
||||||
|
Download --> DB
|
||||||
|
Queue --> Notify --> Discord
|
||||||
|
EventBus -->|"progress"| WS -->|"push"| Browser
|
||||||
|
Download -->|"emit"| EventBus
|
||||||
|
Routes --> DB
|
||||||
|
Scheduler --> DB
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tech Stack
|
||||||
|
|
||||||
|
| Layer | Technology | Purpose |
|
||||||
|
|-------|-----------|---------|
|
||||||
|
| HTTP Server | Fastify 5.2.1 | High-performance, plugin-based web framework |
|
||||||
|
| Frontend | React 19 + Vite | SPA with HMR in development |
|
||||||
|
| Router | React Router 7 | Client-side page navigation |
|
||||||
|
| API Client | TanStack Query 5 | Data fetching, caching, background refetch |
|
||||||
|
| ORM | Drizzle ORM 0.38 | Type-safe SQL queries and schema management |
|
||||||
|
| Database | SQLite via LibSQL | Embedded database with WAL mode |
|
||||||
|
| Migrations | Drizzle Kit | Schema change management |
|
||||||
|
| Downloader | yt-dlp | Media downloading from 1000+ platforms |
|
||||||
|
| Transcoding | ffmpeg | Post-processing, format conversion |
|
||||||
|
| Scheduler | croner | Cron-based channel monitoring |
|
||||||
|
| Icons | Lucide React | UI icon library |
|
||||||
|
| Testing | Vitest | Unit and integration testing |
|
||||||
|
| TypeScript | 5.7.3 | Type safety across full stack |
|
||||||
|
|
||||||
|
## Directory Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
tubearr/
|
||||||
|
├── src/
|
||||||
|
│ ├── index.ts # Application entry point & startup orchestration
|
||||||
|
│ ├── config/
|
||||||
|
│ │ └── index.ts # AppConfig — all env vars parsed here
|
||||||
|
│ ├── db/
|
||||||
|
│ │ ├── index.ts # Database init (WAL mode, foreign keys)
|
||||||
|
│ │ ├── migrate.ts # Migration runner
|
||||||
|
│ │ ├── schema/ # Drizzle table definitions
|
||||||
|
│ │ │ ├── channels.ts # Monitored channels
|
||||||
|
│ │ │ ├── content.ts # Content items + format profiles
|
||||||
|
│ │ │ ├── queue.ts # Download queue
|
||||||
|
│ │ │ ├── history.ts # Activity log
|
||||||
|
│ │ │ ├── notifications.ts # Notification settings
|
||||||
|
│ │ │ ├── platform-settings.ts # Per-platform defaults
|
||||||
|
│ │ │ ├── playlists.ts # Playlists + junction table
|
||||||
|
│ │ │ └── system.ts # Key-value config store
|
||||||
|
│ │ └── repositories/ # Data access layer (one per table)
|
||||||
|
│ ├── server/
|
||||||
|
│ │ ├── index.ts # Fastify builder, plugin registration
|
||||||
|
│ │ ├── middleware/
|
||||||
|
│ │ │ ├── auth.ts # API key + same-origin auth
|
||||||
|
│ │ │ └── error-handler.ts # Centralized error responses
|
||||||
|
│ │ └── routes/ # One file per API resource
|
||||||
|
│ │ ├── channel.ts # Channel CRUD + scan
|
||||||
|
│ │ ├── content.ts # Content listing + filters
|
||||||
|
│ │ ├── download.ts # Enqueue downloads
|
||||||
|
│ │ ├── queue.ts # Queue management
|
||||||
|
│ │ ├── format-profile.ts # Quality presets
|
||||||
|
│ │ ├── notification.ts # Notification config
|
||||||
|
│ │ ├── platform-settings.ts # Platform defaults
|
||||||
|
│ │ ├── history.ts # Activity log
|
||||||
|
│ │ ├── health.ts # GET /ping
|
||||||
|
│ │ ├── scan.ts # Manual channel scan
|
||||||
|
│ │ ├── collect.ts # Back-catalog import
|
||||||
|
│ │ ├── system.ts # API key, settings, yt-dlp
|
||||||
|
│ │ ├── playlist.ts # Playlist data
|
||||||
|
│ │ └── websocket.ts # WebSocket progress stream
|
||||||
|
│ ├── services/ # Business logic layer
|
||||||
|
│ │ ├── scheduler.ts # Cron-based channel monitoring
|
||||||
|
│ │ ├── queue.ts # Download queue + concurrency control
|
||||||
|
│ │ ├── download.ts # Download orchestration (yt-dlp)
|
||||||
|
│ │ ├── notification.ts # Notification dispatch
|
||||||
|
│ │ ├── event-bus.ts # Pub/sub for real-time events
|
||||||
|
│ │ ├── progress-parser.ts # Parse yt-dlp JSON progress
|
||||||
|
│ │ ├── file-organizer.ts # Organize files by platform/channel
|
||||||
|
│ │ ├── quality-analyzer.ts # Post-download codec/resolution check
|
||||||
|
│ │ ├── rate-limiter.ts # Per-platform request throttling
|
||||||
|
│ │ ├── cookie-manager.ts # Platform cookie persistence
|
||||||
|
│ │ ├── health.ts # System health diagnostics
|
||||||
|
│ │ └── back-catalog-import.ts # Bulk media import
|
||||||
|
│ ├── sources/ # Platform adapters
|
||||||
|
│ │ ├── platform-source.ts # Abstract interface
|
||||||
|
│ │ ├── youtube.ts # YouTube channel/content fetcher
|
||||||
|
│ │ ├── soundcloud.ts # SoundCloud artist/track fetcher
|
||||||
|
│ │ ├── generic.ts # Fallback URL handler
|
||||||
|
│ │ └── yt-dlp.ts # yt-dlp subprocess wrapper
|
||||||
|
│ ├── types/
|
||||||
|
│ │ └── index.ts # Shared TypeScript types
|
||||||
|
│ ├── __tests__/ # 40+ Vitest test suites
|
||||||
|
│ └── frontend/
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── App.tsx # React router root
|
||||||
|
│ │ ├── main.tsx # DOM mount
|
||||||
|
│ │ ├── api/ # API client functions
|
||||||
|
│ │ ├── components/ # Reusable UI components
|
||||||
|
│ │ ├── contexts/ # React context (auth, etc.)
|
||||||
|
│ │ ├── hooks/ # Custom hooks (useChannels, etc.)
|
||||||
|
│ │ ├── pages/ # Page-level components
|
||||||
|
│ │ ├── styles/ # CSS
|
||||||
|
│ │ └── utils/ # Frontend utilities
|
||||||
|
│ ├── index.html # SPA template
|
||||||
|
│ ├── vite.config.ts # Vite build config
|
||||||
|
│ └── tsconfig.json # Frontend TS config
|
||||||
|
├── drizzle/ # Generated migration SQL files
|
||||||
|
├── Dockerfile # Multi-stage Docker build
|
||||||
|
├── docker-compose.yml # Compose orchestration
|
||||||
|
├── package.json # Dependencies + scripts
|
||||||
|
├── tsconfig.json # Backend TS config
|
||||||
|
└── vitest.config.ts # Test config
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Design Decisions
|
||||||
|
|
||||||
|
### SQLite over PostgreSQL
|
||||||
|
Tubearr uses embedded SQLite (via LibSQL) instead of a separate database server. This simplifies deployment — no external DB container needed — and is sufficient for a single-user media management app. WAL mode enables concurrent reads during writes.
|
||||||
|
|
||||||
|
### Single-process architecture
|
||||||
|
All services (HTTP, WebSocket, scheduler, queue, downloads) run in one Node.js process. This avoids IPC complexity and keeps the deployment footprint small. The tradeoff is that a crash takes down everything, but the Docker restart policy handles recovery.
|
||||||
|
|
||||||
|
### In-memory queue over external message broker
|
||||||
|
The download queue runs in-process rather than through Redis/RabbitMQ. Queue state is persisted to SQLite so interrupted downloads survive restarts. This is appropriate for the expected throughput (tens of downloads, not thousands).
|
||||||
|
|
||||||
|
### yt-dlp as subprocess
|
||||||
|
Rather than reimplementing platform-specific download logic, Tubearr delegates to yt-dlp via child_process. This provides support for 1000+ platforms, SponsorBlock, subtitle handling, and format selection without maintaining any of that code. The tradeoff is a Python runtime dependency in the Docker image.
|
||||||
|
|
||||||
|
### Same-origin auth bypass
|
||||||
|
Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key.
|
||||||
|
|
||||||
|
### Vite dev middleware in Fastify
|
||||||
|
In development, Vite's middleware is injected into the Fastify server so both backend and frontend run on port 8989 with HMR. In production, the pre-built SPA is served as static files. This avoids CORS issues and simplifies the dev experience.
|
||||||
|
|
||||||
|
## Service Dependencies
|
||||||
|
|
||||||
|
Tubearr has **no external service dependencies** beyond:
|
||||||
|
- **yt-dlp** — installed via pip in the Docker image (system dependency)
|
||||||
|
- **ffmpeg** — installed via apk in the Docker image (system dependency)
|
||||||
|
|
||||||
|
No PostgreSQL, Redis, or other containers required. The Docker image is fully self-contained.
|
||||||
|
|
||||||
|
## Request Lifecycle
|
||||||
|
|
||||||
|
### API Request
|
||||||
|
```
|
||||||
|
Browser/Client → nginx01 (TLS) → Fastify (8989)
|
||||||
|
→ Auth middleware (same-origin check OR API key validation)
|
||||||
|
→ Route handler (validate input, call service/repository)
|
||||||
|
→ Drizzle ORM → SQLite (WAL mode)
|
||||||
|
→ JSON response
|
||||||
|
```
|
||||||
|
|
||||||
|
### Download Flow
|
||||||
|
```
|
||||||
|
User clicks "Download" → POST /api/v1/download/:id
|
||||||
|
→ QueueService.enqueue() → queue_items row (pending)
|
||||||
|
→ QueueService.processNext() picks up item
|
||||||
|
→ RateLimiter.acquire(platform) → wait if throttled
|
||||||
|
→ DownloadService.downloadItem()
|
||||||
|
→ Build yt-dlp args from format profile
|
||||||
|
→ Spawn yt-dlp subprocess
|
||||||
|
→ ProgressParser streams JSON → EventBus → WebSocket → Browser
|
||||||
|
→ FileOrganizer.organize() → move to /media/Platform/Channel/
|
||||||
|
→ QualityAnalyzer.analyze() → inspect codec/resolution
|
||||||
|
→ Update content_items (filePath, fileSize, status=downloaded)
|
||||||
|
→ NotificationService.dispatch(onDownload)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Channel Monitoring
|
||||||
|
```
|
||||||
|
SchedulerService starts cron job per channel (e.g., every 6 hours)
|
||||||
|
→ PlatformSource.fetchRecentContent(channel)
|
||||||
|
→ yt-dlp --flat-playlist --dump-single-json
|
||||||
|
→ Parse response → list of content items
|
||||||
|
→ Deduplicate against existing platformContentIds in DB
|
||||||
|
→ Insert new content_items (status=monitored)
|
||||||
|
→ If monitoringMode includes auto-download → enqueue items
|
||||||
|
→ Update lastCheckedAt, lastCheckStatus
|
||||||
|
```
|
||||||
|
|
||||||
|
## Startup Sequence
|
||||||
|
|
||||||
|
1. Load `.env` via dotenv
|
||||||
|
2. Initialize SQLite with WAL pragmas (`journal_mode`, `busy_timeout`, `foreign_keys`)
|
||||||
|
3. Run Drizzle migrations (idempotent)
|
||||||
|
4. Seed default format profile and system config
|
||||||
|
5. Check/update yt-dlp version (production only)
|
||||||
|
6. Create Vite dev server (development only)
|
||||||
|
7. Build Fastify server — register plugins, middleware, routes
|
||||||
|
8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService
|
||||||
|
9. Register platform sources (YouTube, SoundCloud, Generic)
|
||||||
|
10. Start Fastify on `TUBEARR_PORT`
|
||||||
|
11. Recover interrupted queue items → restart processing
|
||||||
|
12. Start scheduler (monitor enabled channels)
|
||||||
Loading…
Add table
Reference in a new issue