xpltdco/tubearr
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: update Architecture wiki with M004 services (media-server, nfo-generator, keyword-filter, missing-file-scanner, feed, pause/resume, ad-hoc download flow)

xpltd_admin 2026-04-04 02:07:18 -06:00
parent 380a1b0c32
commit 697d74d7e8
1 changed files with 89 additions and 39 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

128
Architecture.md

@ -28,28 +28,38 @@ graph TB
FileSystem["Media Files (/media)"]
Notify["Notification Service"]
Discord["Discord Webhook"]
MediaServer["Media Servers (Plex/Jellyfin)"]
NfoGen["NFO Generator"]
MissingScan["Missing File Scanner"]
KeywordFilter["Keyword Filter"]
RSS["RSS Feed"]
Browser -->|HTTP/WS| Fastify
Fastify --> Auth --> Routes
Fastify --> WS
Routes --> Queue
Routes --> Scheduler
Routes --> RSS
Scheduler -->|"cron jobs"| Queue
Scheduler -->|"filter"| KeywordFilter
Queue -->|"process"| Download
Download --> YtDlp
YtDlp --> FileSystem
Download --> DB
Download -->|"sidecar"| NfoGen --> FileSystem
Queue --> Notify --> Discord
EventBus -->|"progress"| WS -->|"push"| Browser
Download -->|"emit"| EventBus
EventBus -->|"download:complete"| MediaServer
Routes --> DB
Routes --> MissingScan --> DB
Scheduler --> DB
```
## Tech Stack
| Layer | Technology | Purpose |
|-------|-----------|---------|
|-------|-----------|--------|
| HTTP Server | Fastify 5.2.1 | High-performance, plugin-based web framework |
| Frontend | React 19 + Vite | SPA with HMR in development |
| Router | React Router 7 | Client-side page navigation |
@ -61,7 +71,7 @@ graph TB
| Transcoding | ffmpeg | Post-processing, format conversion |
| Scheduler | croner | Cron-based channel monitoring |
| Icons | Lucide React | UI icon library |
| Testing | Vitest | Unit and integration testing |
| Testing | Vitest | Unit and integration testing (830+ tests) |
| TypeScript | 5.7.3 | Type safety across full stack |
## Directory Structure
@ -76,72 +86,81 @@ tubearr/
│ │ ├── index.ts # Database init (WAL mode, foreign keys)
│ │ ├── migrate.ts # Migration runner
│ │ ├── schema/ # Drizzle table definitions
│ │ │ ├── channels.ts # Monitored channels
│ │ │ ├── content.ts # Content items + format profiles
│ │ │ ├── queue.ts # Download queue
│ │ │ ├── channels.ts # Monitored channels (+ keyword filter columns)
│ │ │ ├── content.ts # Content items + format profiles (+ outputTemplate, contentRating)
│ │ │ ├── queue.ts # Download queue (+ paused status)
│ │ │ ├── history.ts # Activity log
│ │ │ ├── notifications.ts # Notification settings
│ │ │ ├── platform-settings.ts # Per-platform defaults
│ │ │ ├── playlists.ts # Playlists + junction table
│ │ │ ├── media-servers.ts # Plex/Jellyfin server configs
│ │ │ └── system.ts # Key-value config store
│ │ └── repositories/ # Data access layer (one per table)
│ ├── server/
│ │ ├── index.ts # Fastify builder, plugin registration
│ │ ├── middleware/
│ │ │ ├── auth.ts # API key + same-origin auth
│ │ │ ├── auth.ts # API key + same-origin auth (public paths: /ping, /feed/rss, /media)
│ │ │ └── error-handler.ts # Centralized error responses
│ │ └── routes/ # One file per API resource
│ │ ├── channel.ts # Channel CRUD + scan
│ │ ├── content.ts # Content listing + filters
│ │ ├── channel.ts # Channel CRUD + keyword filter updates
│ │ ├── content.ts # Content listing + filters + rating updates + content type counts
│ │ ├── download.ts # Enqueue downloads
│ │ ├── queue.ts # Queue management
│ │ ├── format-profile.ts # Quality presets
│ │ ├── adhoc-download.ts # Ad-hoc URL preview + confirm endpoints
│ │ ├── queue.ts # Queue management + pause/resume
│ │ ├── format-profile.ts # Quality presets + output templates
│ │ ├── notification.ts # Notification config
│ │ ├── platform-settings.ts # Platform defaults
│ │ ├── history.ts # Activity log
│ │ ├── health.ts # GET /ping
│ │ ├── scan.ts # Manual channel scan
│ │ ├── scan.ts # Manual channel scan (fire-and-forget + scan-all)
│ │ ├── collect.ts # Back-catalog import
│ │ ├── system.ts # API key, settings, yt-dlp
│ │ ├── system.ts # Settings, yt-dlp, missing file scan triggers
│ │ ├── media-server.ts # Media server CRUD + test + sections
│ │ ├── feed.ts # RSS 2.0 podcast feed + media streaming
│ │ ├── playlist.ts # Playlist data
│ │ └── websocket.ts # WebSocket progress stream
│ ├── services/ # Business logic layer
│ │ ├── scheduler.ts # Cron-based channel monitoring
│ │ ├── queue.ts # Download queue + concurrency control
│ │ ├── download.ts # Download orchestration (yt-dlp)
│ │ ├── scheduler.ts # Cron-based channel monitoring + keyword filter integration
│ │ ├── queue.ts # Download queue + concurrency + pause/resume via AbortController
│ │ ├── download.ts # Download orchestration (yt-dlp) + NFO sidecar generation
│ │ ├── notification.ts # Notification dispatch
│ │ ├── event-bus.ts # Pub/sub for real-time events
│ │ ├── event-bus.ts # Pub/sub for real-time events + download:complete for media servers
│ │ ├── progress-parser.ts # Parse yt-dlp JSON progress
│ │ ├── file-organizer.ts # Organize files by platform/channel
│ │ ├── file-organizer.ts # Organize files + output path template resolution
│ │ ├── keyword-filter.ts # Per-channel include/exclude pattern matching (plain, glob, regex)
│ │ ├── media-server.ts # Stateless Plex/Jellyfin scan, test, sections operations
│ │ ├── nfo-generator.ts # Kodi-compatible NFO XML sidecar generation
│ │ ├── missing-file-scanner.ts # Cursor-based batch filesystem existence checking
│ │ ├── quality-analyzer.ts # Post-download codec/resolution check
│ │ ├── rate-limiter.ts # Per-platform request throttling
│ │ ├── cookie-manager.ts # Platform cookie persistence
│ │ ├── health.ts # System health diagnostics
│ │ └── back-catalog-import.ts # Bulk media import
│ ├── sources/ # Platform adapters
│ │ ├── platform-source.ts # Abstract interface
│ │ ├── youtube.ts # YouTube channel/content fetcher
│ │ ├── platform-source.ts # Abstract interface + URL detection (YouTube, SoundCloud, Generic)
│ │ ├── youtube.ts # YouTube channel/content/playlist fetcher
│ │ ├── soundcloud.ts # SoundCloud artist/track fetcher
│ │ ├── generic.ts # Fallback URL handler
│ │ ├── generic.ts # Fallback URL handler (any yt-dlp-supported site)
│ │ └── yt-dlp.ts # yt-dlp subprocess wrapper
│ ├── types/
│ │ └── index.ts # Shared TypeScript types
│ ├── __tests__/ # 40+ Vitest test suites
│ │ ├── index.ts # Shared TypeScript types
│ │ └── api.ts # API response types
│ ├── __tests__/ # 46 Vitest test suites, 830+ tests
│ └── frontend/
│ ├── src/
│ │ ├── App.tsx # React router root
│ │ ├── main.tsx # DOM mount
│ │ ├── api/ # API client functions
│ │ ├── components/ # Reusable UI components
│ │ ├── contexts/ # React context (auth, etc.)
│ │ ├── hooks/ # Custom hooks (useChannels, etc.)
│ │ ├── pages/ # Page-level components
│ │ ├── api/ # API client functions + TanStack Query hooks
│ │ ├── components/ # Reusable UI (AddUrlModal, MediaServerForm, RatingBadge, StatusBadge, ...)
│ │ ├── contexts/ # React context (auth, download progress, scan state)
│ │ ├── hooks/ # Custom hooks (useTimezone, useTheme, useChannels, ...)
│ │ ├── pages/ # Page-level components (Library, ChannelDetail, Settings, System, ...)
│ │ ├── styles/ # CSS
│ │ └── utils/ # Frontend utilities
│ ├── index.html # SPA template
│ │ └── utils/ # Frontend utilities (format.ts for timezone-aware formatting)
│ ├── index.html # SPA template (inline theme script for flash prevention)
│ ├── vite.config.ts # Vite build config
│ └── tsconfig.json # Frontend TS config
├── drizzle/ # Generated migration SQL files
├── drizzle/ # Generated migration SQL files (0001–0017)
├── Dockerfile # Multi-stage Docker build
├── docker-compose.yml # Compose orchestration
├── package.json # Dependencies + scripts
@ -164,7 +183,19 @@ The download queue runs in-process rather than through Redis/RabbitMQ. Queue sta
Rather than reimplementing platform-specific download logic, Tubearr delegates to yt-dlp via child_process. This provides support for 1000+ platforms, SponsorBlock, subtitle handling, and format selection without maintaining any of that code. The tradeoff is a Python runtime dependency in the Docker image.
### Same-origin auth bypass
Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key.
Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key. Public paths (`/ping`, `/api/v1/feed/rss`, `/api/v1/media/:id/:filename`) bypass auth entirely.
### Best-effort sidecar pattern
Post-download auxiliary operations (NFO generation, media server scan triggers) are wrapped in try/catch so they never fail the primary download. The download is the primary value; sidecars are additive. Failures are logged for debugging but don't propagate.
### Fire-and-forget media server scans
On download:complete, scans are triggered across all enabled media servers via Promise.allSettled. One failing server doesn't block others or the download pipeline.
### Dual persistence for visual settings
Theme is stored in both localStorage (read by an inline script before React hydrates) and the database (survives server restarts). This prevents flash-of-wrong-theme on page load.
### AbortController for download pause/resume
Pause aborts the in-flight yt-dlp subprocess without incrementing the attempt counter. Resume resets to pending and triggers processNext. This means pause/resume is free from the retry budget perspective.
### Vite dev middleware in Fastify
In development, Vite's middleware is injected into the Fastify server so both backend and frontend run on port 8989 with HMR. In production, the pre-built SPA is served as static files. This avoids CORS issues and simplifies the dev experience.
@ -175,14 +206,18 @@ Tubearr has **no external service dependencies** beyond:
- **yt-dlp** — installed via pip in the Docker image (system dependency)
- **ffmpeg** — installed via apk in the Docker image (system dependency)
Optional integrations:
- **Plex** — auto-scan library on download completion
- **Jellyfin** — auto-scan library on download completion
No PostgreSQL, Redis, or other containers required. The Docker image is fully self-contained.
## Request Lifecycle
### API Request
```
Browser/Client → nginx01 (TLS) → Fastify (8989)
→ Auth middleware (same-origin check OR API key validation)
Browser/Client → nginx01 (TLS) → Fastify (8097)
→ Auth middleware (same-origin check OR API key validation OR public path bypass)
→ Route handler (validate input, call service/repository)
→ Drizzle ORM → SQLite (WAL mode)
→ JSON response
@ -196,14 +231,27 @@ User clicks "Download" → POST /api/v1/download/:id
→ RateLimiter.acquire(platform) → wait if throttled
→ DownloadService.downloadItem()
→ Build yt-dlp args from format profile
→ Spawn yt-dlp subprocess
→ Spawn yt-dlp subprocess (cancellable via AbortController)
→ ProgressParser streams JSON → EventBus → WebSocket → Browser
→ FileOrganizer.organize() → move to /media/Platform/Channel/
→ FileOrganizer.organize() → resolve output template → move to /media/...
→ NfoGenerator.writeNfoFile() → Kodi XML sidecar (best-effort)
→ QualityAnalyzer.analyze() → inspect codec/resolution
→ Update content_items (filePath, fileSize, status=downloaded)
→ EventBus.emit('download:complete') → MediaServerService.scanAll() (fire-and-forget)
→ NotificationService.dispatch(onDownload)
```
### Ad-hoc URL Download Flow
```
User pastes URL → POST /api/v1/download/url/preview
→ yt-dlp --dump-json --no-download --no-playlist
→ Return metadata preview (title, thumbnail, duration, platform)
User confirms → POST /api/v1/download/url/confirm
→ Create content_item (channelId=null)
→ Enqueue for download
→ Same download flow as above
```
### Channel Monitoring
```
SchedulerService starts cron job per channel (e.g., every 6 hours)
@ -211,6 +259,7 @@ SchedulerService starts cron job per channel (e.g., every 6 hours)
→ yt-dlp --flat-playlist --dump-single-json
→ Parse response → list of content items
→ Deduplicate against existing platformContentIds in DB
→ KeywordFilter.matchesKeywordFilter() → apply include/exclude patterns
→ Insert new content_items (status=monitored)
→ If monitoringMode includes auto-download → enqueue items
→ Update lastCheckedAt, lastCheckStatus
@ -225,8 +274,9 @@ SchedulerService starts cron job per channel (e.g., every 6 hours)
5. Check/update yt-dlp version (production only)
6. Create Vite dev server (development only)
7. Build Fastify server — register plugins, middleware, routes
8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService
8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService → MediaServerService → MissingFileScanner → NfoGenerator
9. Register platform sources (YouTube, SoundCloud, Generic)
10. Start Fastify on `TUBEARR_PORT`
11. Recover interrupted queue items → restart processing
12. Start scheduler (monitor enabled channels)
10. Wire EventBus download:complete → media server auto-scan
11. Start Fastify on `TUBEARR_PORT`
12. Recover interrupted queue items → restart processing
13. Start scheduler (monitor enabled channels)