2 Architecture

xpltd_admin edited this page 2026-04-04 02:07:18 -06:00

Unescape Escape

Table of Contents

• Architecture
• System Overview
• Tech Stack
• Directory Structure
• Key Design Decisions
• SQLite over PostgreSQL
• Single-process architecture
• In-memory queue over external message broker
• yt-dlp as subprocess
• Same-origin auth bypass
• Best-effort sidecar pattern
• Fire-and-forget media server scans
• Dual persistence for visual settings
• AbortController for download pause/resume
• Vite dev middleware in Fastify
• Service Dependencies
• Request Lifecycle
• API Request
• Download Flow
• Ad-hoc URL Download Flow
• Channel Monitoring
• Startup Sequence

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Architecture

Meta	Value
Repo	xpltdco/tubearr
Page	Architecture
Audience	developers, agents, newcomers
Last Updated	2026-04-04
Status	current

System Overview

Tubearr is a single-process Node.js application with an embedded SQLite database. The backend (Fastify) serves both the REST API and the React SPA. Downloads are performed by spawning yt-dlp subprocesses. Real-time progress is streamed to the browser via WebSocket.

graph TB
    Browser["Browser (React SPA)"]
    Fastify["Fastify Server"]
    Auth["Auth Middleware"]
    Routes["API Routes"]
    WS["WebSocket /ws"]
    Scheduler["Scheduler Service"]
    Queue["Queue Service"]
    Download["Download Service"]
    EventBus["Event Bus"]
    YtDlp["yt-dlp subprocess"]
    DB["SQLite (LibSQL)"]
    FileSystem["Media Files (/media)"]
    Notify["Notification Service"]
    Discord["Discord Webhook"]
    MediaServer["Media Servers (Plex/Jellyfin)"]
    NfoGen["NFO Generator"]
    MissingScan["Missing File Scanner"]
    KeywordFilter["Keyword Filter"]
    RSS["RSS Feed"]

    Browser -->|HTTP/WS| Fastify
    Fastify --> Auth --> Routes
    Fastify --> WS
    Routes --> Queue
    Routes --> Scheduler
    Routes --> RSS
    Scheduler -->|"cron jobs"| Queue
    Scheduler -->|"filter"| KeywordFilter
    Queue -->|"process"| Download
    Download --> YtDlp
    YtDlp --> FileSystem
    Download --> DB
    Download -->|"sidecar"| NfoGen --> FileSystem
    Queue --> Notify --> Discord
    EventBus -->|"progress"| WS -->|"push"| Browser
    Download -->|"emit"| EventBus
    EventBus -->|"download:complete"| MediaServer
    Routes --> DB
    Routes --> MissingScan --> DB
    Scheduler --> DB

Tech Stack

Layer	Technology	Purpose
HTTP Server	Fastify 5.2.1	High-performance, plugin-based web framework
Frontend	React 19 + Vite	SPA with HMR in development
Router	React Router 7	Client-side page navigation
API Client	TanStack Query 5	Data fetching, caching, background refetch
ORM	Drizzle ORM 0.38	Type-safe SQL queries and schema management
Database	SQLite via LibSQL	Embedded database with WAL mode
Migrations	Drizzle Kit	Schema change management
Downloader	yt-dlp	Media downloading from 1000+ platforms
Transcoding	ffmpeg	Post-processing, format conversion
Scheduler	croner	Cron-based channel monitoring
Icons	Lucide React	UI icon library
Testing	Vitest	Unit and integration testing (830+ tests)
TypeScript	5.7.3	Type safety across full stack

Directory Structure

tubearr/
├── src/
│   ├── index.ts                    # Application entry point & startup orchestration
│   ├── config/
│   │   └── index.ts                # AppConfig — all env vars parsed here
│   ├── db/
│   │   ├── index.ts                # Database init (WAL mode, foreign keys)
│   │   ├── migrate.ts              # Migration runner
│   │   ├── schema/                 # Drizzle table definitions
│   │   │   ├── channels.ts         # Monitored channels (+ keyword filter columns)
│   │   │   ├── content.ts          # Content items + format profiles (+ outputTemplate, contentRating)
│   │   │   ├── queue.ts            # Download queue (+ paused status)
│   │   │   ├── history.ts          # Activity log
│   │   │   ├── notifications.ts    # Notification settings
│   │   │   ├── platform-settings.ts # Per-platform defaults
│   │   │   ├── playlists.ts        # Playlists + junction table
│   │   │   ├── media-servers.ts    # Plex/Jellyfin server configs
│   │   │   └── system.ts           # Key-value config store
│   │   └── repositories/           # Data access layer (one per table)
│   ├── server/
│   │   ├── index.ts                # Fastify builder, plugin registration
│   │   ├── middleware/
│   │   │   ├── auth.ts             # API key + same-origin auth (public paths: /ping, /feed/rss, /media)
│   │   │   └── error-handler.ts    # Centralized error responses
│   │   └── routes/                 # One file per API resource
│   │       ├── channel.ts          # Channel CRUD + keyword filter updates
│   │       ├── content.ts          # Content listing + filters + rating updates + content type counts
│   │       ├── download.ts         # Enqueue downloads
│   │       ├── adhoc-download.ts   # Ad-hoc URL preview + confirm endpoints
│   │       ├── queue.ts            # Queue management + pause/resume
│   │       ├── format-profile.ts   # Quality presets + output templates
│   │       ├── notification.ts     # Notification config
│   │       ├── platform-settings.ts # Platform defaults
│   │       ├── history.ts          # Activity log
│   │       ├── health.ts           # GET /ping
│   │       ├── scan.ts             # Manual channel scan (fire-and-forget + scan-all)
│   │       ├── collect.ts          # Back-catalog import
│   │       ├── system.ts           # Settings, yt-dlp, missing file scan triggers
│   │       ├── media-server.ts     # Media server CRUD + test + sections
│   │       ├── feed.ts             # RSS 2.0 podcast feed + media streaming
│   │       ├── playlist.ts         # Playlist data
│   │       └── websocket.ts        # WebSocket progress stream
│   ├── services/                   # Business logic layer
│   │   ├── scheduler.ts            # Cron-based channel monitoring + keyword filter integration
│   │   ├── queue.ts                # Download queue + concurrency + pause/resume via AbortController
│   │   ├── download.ts             # Download orchestration (yt-dlp) + NFO sidecar generation
│   │   ├── notification.ts         # Notification dispatch
│   │   ├── event-bus.ts            # Pub/sub for real-time events + download:complete for media servers
│   │   ├── progress-parser.ts      # Parse yt-dlp JSON progress
│   │   ├── file-organizer.ts       # Organize files + output path template resolution
│   │   ├── keyword-filter.ts       # Per-channel include/exclude pattern matching (plain, glob, regex)
│   │   ├── media-server.ts         # Stateless Plex/Jellyfin scan, test, sections operations
│   │   ├── nfo-generator.ts        # Kodi-compatible NFO XML sidecar generation
│   │   ├── missing-file-scanner.ts # Cursor-based batch filesystem existence checking
│   │   ├── quality-analyzer.ts     # Post-download codec/resolution check
│   │   ├── rate-limiter.ts         # Per-platform request throttling
│   │   ├── cookie-manager.ts       # Platform cookie persistence
│   │   ├── health.ts               # System health diagnostics
│   │   └── back-catalog-import.ts  # Bulk media import
│   ├── sources/                    # Platform adapters
│   │   ├── platform-source.ts      # Abstract interface + URL detection (YouTube, SoundCloud, Generic)
│   │   ├── youtube.ts              # YouTube channel/content/playlist fetcher
│   │   ├── soundcloud.ts           # SoundCloud artist/track fetcher
│   │   ├── generic.ts              # Fallback URL handler (any yt-dlp-supported site)
│   │   └── yt-dlp.ts               # yt-dlp subprocess wrapper
│   ├── types/
│   │   ├── index.ts                # Shared TypeScript types
│   │   └── api.ts                  # API response types
│   ├── __tests__/                  # 46 Vitest test suites, 830+ tests
│   └── frontend/
│       ├── src/
│       │   ├── App.tsx             # React router root
│       │   ├── main.tsx            # DOM mount
│       │   ├── api/                # API client functions + TanStack Query hooks
│       │   ├── components/         # Reusable UI (AddUrlModal, MediaServerForm, RatingBadge, StatusBadge, ...)
│       │   ├── contexts/           # React context (auth, download progress, scan state)
│       │   ├── hooks/              # Custom hooks (useTimezone, useTheme, useChannels, ...)
│       │   ├── pages/              # Page-level components (Library, ChannelDetail, Settings, System, ...)
│       │   ├── styles/             # CSS
│       │   └── utils/              # Frontend utilities (format.ts for timezone-aware formatting)
│       ├── index.html              # SPA template (inline theme script for flash prevention)
│       ├── vite.config.ts          # Vite build config
│       └── tsconfig.json           # Frontend TS config
├── drizzle/                        # Generated migration SQL files (0001–0017)
├── Dockerfile                      # Multi-stage Docker build
├── docker-compose.yml              # Compose orchestration
├── package.json                    # Dependencies + scripts
├── tsconfig.json                   # Backend TS config
└── vitest.config.ts                # Test config

Key Design Decisions

SQLite over PostgreSQL

Tubearr uses embedded SQLite (via LibSQL) instead of a separate database server. This simplifies deployment — no external DB container needed — and is sufficient for a single-user media management app. WAL mode enables concurrent reads during writes.

Single-process architecture

All services (HTTP, WebSocket, scheduler, queue, downloads) run in one Node.js process. This avoids IPC complexity and keeps the deployment footprint small. The tradeoff is that a crash takes down everything, but the Docker restart policy handles recovery.

In-memory queue over external message broker

The download queue runs in-process rather than through Redis/RabbitMQ. Queue state is persisted to SQLite so interrupted downloads survive restarts. This is appropriate for the expected throughput (tens of downloads, not thousands).

yt-dlp as subprocess

Rather than reimplementing platform-specific download logic, Tubearr delegates to yt-dlp via child_process. This provides support for 1000+ platforms, SponsorBlock, subtitle handling, and format selection without maintaining any of that code. The tradeoff is a Python runtime dependency in the Docker image.

Same-origin auth bypass

Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key. Public paths (/ping, /api/v1/feed/rss, /api/v1/media/:id/:filename) bypass auth entirely.

Best-effort sidecar pattern

Post-download auxiliary operations (NFO generation, media server scan triggers) are wrapped in try/catch so they never fail the primary download. The download is the primary value; sidecars are additive. Failures are logged for debugging but don't propagate.

Fire-and-forget media server scans

On download:complete, scans are triggered across all enabled media servers via Promise.allSettled. One failing server doesn't block others or the download pipeline.

Dual persistence for visual settings

Theme is stored in both localStorage (read by an inline script before React hydrates) and the database (survives server restarts). This prevents flash-of-wrong-theme on page load.

AbortController for download pause/resume

Pause aborts the in-flight yt-dlp subprocess without incrementing the attempt counter. Resume resets to pending and triggers processNext. This means pause/resume is free from the retry budget perspective.

Vite dev middleware in Fastify

In development, Vite's middleware is injected into the Fastify server so both backend and frontend run on port 8989 with HMR. In production, the pre-built SPA is served as static files. This avoids CORS issues and simplifies the dev experience.

Service Dependencies

Tubearr has no external service dependencies beyond:

• yt-dlp — installed via pip in the Docker image (system dependency)
• ffmpeg — installed via apk in the Docker image (system dependency)

Optional integrations:

• Plex — auto-scan library on download completion
• Jellyfin — auto-scan library on download completion

No PostgreSQL, Redis, or other containers required. The Docker image is fully self-contained.

Request Lifecycle

API Request

Browser/Client → nginx01 (TLS) → Fastify (8097)
  → Auth middleware (same-origin check OR API key validation OR public path bypass)
  → Route handler (validate input, call service/repository)
  → Drizzle ORM → SQLite (WAL mode)
  → JSON response

Download Flow

User clicks "Download" → POST /api/v1/download/:id
  → QueueService.enqueue() → queue_items row (pending)
  → QueueService.processNext() picks up item
  → RateLimiter.acquire(platform) → wait if throttled
  → DownloadService.downloadItem()
    → Build yt-dlp args from format profile
    → Spawn yt-dlp subprocess (cancellable via AbortController)
    → ProgressParser streams JSON → EventBus → WebSocket → Browser
    → FileOrganizer.organize() → resolve output template → move to /media/...
    → NfoGenerator.writeNfoFile() → Kodi XML sidecar (best-effort)
    → QualityAnalyzer.analyze() → inspect codec/resolution
    → Update content_items (filePath, fileSize, status=downloaded)
  → EventBus.emit('download:complete') → MediaServerService.scanAll() (fire-and-forget)
  → NotificationService.dispatch(onDownload)

Ad-hoc URL Download Flow

User pastes URL → POST /api/v1/download/url/preview
  → yt-dlp --dump-json --no-download --no-playlist
  → Return metadata preview (title, thumbnail, duration, platform)
User confirms → POST /api/v1/download/url/confirm
  → Create content_item (channelId=null)
  → Enqueue for download
  → Same download flow as above

Channel Monitoring

SchedulerService starts cron job per channel (e.g., every 6 hours)
  → PlatformSource.fetchRecentContent(channel)
    → yt-dlp --flat-playlist --dump-single-json
    → Parse response → list of content items
  → Deduplicate against existing platformContentIds in DB
  → KeywordFilter.matchesKeywordFilter() → apply include/exclude patterns
  → Insert new content_items (status=monitored)
  → If monitoringMode includes auto-download → enqueue items
  → Update lastCheckedAt, lastCheckStatus

Startup Sequence

1. Load .env via dotenv
2. Initialize SQLite with WAL pragmas (journal_mode, busy_timeout, foreign_keys)
3. Run Drizzle migrations (idempotent)
4. Seed default format profile and system config
5. Check/update yt-dlp version (production only)
6. Create Vite dev server (development only)
7. Build Fastify server — register plugins, middleware, routes
8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService → MediaServerService → MissingFileScanner → NfoGenerator
9. Register platform sources (YouTube, SoundCloud, Generic)
10. Wire EventBus download:complete → media server auto-scan
11. Start Fastify on TUBEARR_PORT
12. Recover interrupted queue items → restart processing
13. Start scheduler (monitor enabled channels)