xpltdco/tubearr
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: update Architecture wiki with M004 services (media-server, nfo-generator, keyword-filter, missing-file-scanner, feed, pause/resume, ad-hoc download flow)

xpltd_admin 2026-04-04 02:07:18 -06:00
parent 380a1b0c32
commit 697d74d7e8
1 changed files with 89 additions and 39 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

128
Architecture.md

@ -28,28 +28,38 @@ graph TB
FileSystem["Media Files (/media)"] FileSystem["Media Files (/media)"]
Notify["Notification Service"] Notify["Notification Service"]
Discord["Discord Webhook"] Discord["Discord Webhook"]
MediaServer["Media Servers (Plex/Jellyfin)"]
NfoGen["NFO Generator"]
MissingScan["Missing File Scanner"]
KeywordFilter["Keyword Filter"]
RSS["RSS Feed"]
Browser -->|HTTP/WS| Fastify Browser -->|HTTP/WS| Fastify
Fastify --> Auth --> Routes Fastify --> Auth --> Routes
Fastify --> WS Fastify --> WS
Routes --> Queue Routes --> Queue
Routes --> Scheduler Routes --> Scheduler
Routes --> RSS
Scheduler -->|"cron jobs"| Queue Scheduler -->|"cron jobs"| Queue
Scheduler -->|"filter"| KeywordFilter
Queue -->|"process"| Download Queue -->|"process"| Download
Download --> YtDlp Download --> YtDlp
YtDlp --> FileSystem YtDlp --> FileSystem
Download --> DB Download --> DB
Download -->|"sidecar"| NfoGen --> FileSystem
Queue --> Notify --> Discord Queue --> Notify --> Discord
EventBus -->|"progress"| WS -->|"push"| Browser EventBus -->|"progress"| WS -->|"push"| Browser
Download -->|"emit"| EventBus Download -->|"emit"| EventBus
EventBus -->|"download:complete"| MediaServer
Routes --> DB Routes --> DB
Routes --> MissingScan --> DB
Scheduler --> DB Scheduler --> DB
``` ```
## Tech Stack ## Tech Stack
| Layer | Technology | Purpose | | Layer | Technology | Purpose |
|-------|-----------|---------| |-------|-----------|--------|
| HTTP Server | Fastify 5.2.1 | High-performance, plugin-based web framework | | HTTP Server | Fastify 5.2.1 | High-performance, plugin-based web framework |
| Frontend | React 19 + Vite | SPA with HMR in development | | Frontend | React 19 + Vite | SPA with HMR in development |
| Router | React Router 7 | Client-side page navigation | | Router | React Router 7 | Client-side page navigation |
@ -61,7 +71,7 @@ graph TB
| Transcoding | ffmpeg | Post-processing, format conversion | | Transcoding | ffmpeg | Post-processing, format conversion |
| Scheduler | croner | Cron-based channel monitoring | | Scheduler | croner | Cron-based channel monitoring |
| Icons | Lucide React | UI icon library | | Icons | Lucide React | UI icon library |
| Testing | Vitest | Unit and integration testing | | Testing | Vitest | Unit and integration testing (830+ tests) |
| TypeScript | 5.7.3 | Type safety across full stack | | TypeScript | 5.7.3 | Type safety across full stack |
## Directory Structure ## Directory Structure
@ -76,72 +86,81 @@ tubearr/
│ │ ├── index.ts # Database init (WAL mode, foreign keys) │ │ ├── index.ts # Database init (WAL mode, foreign keys)
│ │ ├── migrate.ts # Migration runner │ │ ├── migrate.ts # Migration runner
│ │ ├── schema/ # Drizzle table definitions │ │ ├── schema/ # Drizzle table definitions
│ │ │ ├── channels.ts # Monitored channels │ │ │ ├── channels.ts # Monitored channels (+ keyword filter columns)
│ │ │ ├── content.ts # Content items + format profiles │ │ │ ├── content.ts # Content items + format profiles (+ outputTemplate, contentRating)
│ │ │ ├── queue.ts # Download queue │ │ │ ├── queue.ts # Download queue (+ paused status)
│ │ │ ├── history.ts # Activity log │ │ │ ├── history.ts # Activity log
│ │ │ ├── notifications.ts # Notification settings │ │ │ ├── notifications.ts # Notification settings
│ │ │ ├── platform-settings.ts # Per-platform defaults │ │ │ ├── platform-settings.ts # Per-platform defaults
│ │ │ ├── playlists.ts # Playlists + junction table │ │ │ ├── playlists.ts # Playlists + junction table
│ │ │ ├── media-servers.ts # Plex/Jellyfin server configs
│ │ │ └── system.ts # Key-value config store │ │ │ └── system.ts # Key-value config store
│ │ └── repositories/ # Data access layer (one per table) │ │ └── repositories/ # Data access layer (one per table)
│ ├── server/ │ ├── server/
│ │ ├── index.ts # Fastify builder, plugin registration │ │ ├── index.ts # Fastify builder, plugin registration
│ │ ├── middleware/ │ │ ├── middleware/
│ │ │ ├── auth.ts # API key + same-origin auth │ │ │ ├── auth.ts # API key + same-origin auth (public paths: /ping, /feed/rss, /media)
│ │ │ └── error-handler.ts # Centralized error responses │ │ │ └── error-handler.ts # Centralized error responses
│ │ └── routes/ # One file per API resource │ │ └── routes/ # One file per API resource
│ │ ├── channel.ts # Channel CRUD + scan │ │ ├── channel.ts # Channel CRUD + keyword filter updates
│ │ ├── content.ts # Content listing + filters │ │ ├── content.ts # Content listing + filters + rating updates + content type counts
│ │ ├── download.ts # Enqueue downloads │ │ ├── download.ts # Enqueue downloads
│ │ ├── queue.ts # Queue management │ │ ├── adhoc-download.ts # Ad-hoc URL preview + confirm endpoints
│ │ ├── format-profile.ts # Quality presets │ │ ├── queue.ts # Queue management + pause/resume
│ │ ├── format-profile.ts # Quality presets + output templates
│ │ ├── notification.ts # Notification config │ │ ├── notification.ts # Notification config
│ │ ├── platform-settings.ts # Platform defaults │ │ ├── platform-settings.ts # Platform defaults
│ │ ├── history.ts # Activity log │ │ ├── history.ts # Activity log
│ │ ├── health.ts # GET /ping │ │ ├── health.ts # GET /ping
│ │ ├── scan.ts # Manual channel scan │ │ ├── scan.ts # Manual channel scan (fire-and-forget + scan-all)
│ │ ├── collect.ts # Back-catalog import │ │ ├── collect.ts # Back-catalog import
│ │ ├── system.ts # API key, settings, yt-dlp │ │ ├── system.ts # Settings, yt-dlp, missing file scan triggers
│ │ ├── media-server.ts # Media server CRUD + test + sections
│ │ ├── feed.ts # RSS 2.0 podcast feed + media streaming
│ │ ├── playlist.ts # Playlist data │ │ ├── playlist.ts # Playlist data
│ │ └── websocket.ts # WebSocket progress stream │ │ └── websocket.ts # WebSocket progress stream
│ ├── services/ # Business logic layer │ ├── services/ # Business logic layer
│ │ ├── scheduler.ts # Cron-based channel monitoring │ │ ├── scheduler.ts # Cron-based channel monitoring + keyword filter integration
│ │ ├── queue.ts # Download queue + concurrency control │ │ ├── queue.ts # Download queue + concurrency + pause/resume via AbortController
│ │ ├── download.ts # Download orchestration (yt-dlp) │ │ ├── download.ts # Download orchestration (yt-dlp) + NFO sidecar generation
│ │ ├── notification.ts # Notification dispatch │ │ ├── notification.ts # Notification dispatch
│ │ ├── event-bus.ts # Pub/sub for real-time events │ │ ├── event-bus.ts # Pub/sub for real-time events + download:complete for media servers
│ │ ├── progress-parser.ts # Parse yt-dlp JSON progress │ │ ├── progress-parser.ts # Parse yt-dlp JSON progress
│ │ ├── file-organizer.ts # Organize files by platform/channel │ │ ├── file-organizer.ts # Organize files + output path template resolution
│ │ ├── keyword-filter.ts # Per-channel include/exclude pattern matching (plain, glob, regex)
│ │ ├── media-server.ts # Stateless Plex/Jellyfin scan, test, sections operations
│ │ ├── nfo-generator.ts # Kodi-compatible NFO XML sidecar generation
│ │ ├── missing-file-scanner.ts # Cursor-based batch filesystem existence checking
│ │ ├── quality-analyzer.ts # Post-download codec/resolution check │ │ ├── quality-analyzer.ts # Post-download codec/resolution check
│ │ ├── rate-limiter.ts # Per-platform request throttling │ │ ├── rate-limiter.ts # Per-platform request throttling
│ │ ├── cookie-manager.ts # Platform cookie persistence │ │ ├── cookie-manager.ts # Platform cookie persistence
│ │ ├── health.ts # System health diagnostics │ │ ├── health.ts # System health diagnostics
│ │ └── back-catalog-import.ts # Bulk media import │ │ └── back-catalog-import.ts # Bulk media import
│ ├── sources/ # Platform adapters │ ├── sources/ # Platform adapters
│ │ ├── platform-source.ts # Abstract interface │ │ ├── platform-source.ts # Abstract interface + URL detection (YouTube, SoundCloud, Generic)
│ │ ├── youtube.ts # YouTube channel/content fetcher │ │ ├── youtube.ts # YouTube channel/content/playlist fetcher
│ │ ├── soundcloud.ts # SoundCloud artist/track fetcher │ │ ├── soundcloud.ts # SoundCloud artist/track fetcher
│ │ ├── generic.ts # Fallback URL handler │ │ ├── generic.ts # Fallback URL handler (any yt-dlp-supported site)
│ │ └── yt-dlp.ts # yt-dlp subprocess wrapper │ │ └── yt-dlp.ts # yt-dlp subprocess wrapper
│ ├── types/ │ ├── types/
│ │ └── index.ts # Shared TypeScript types │ │ ├── index.ts # Shared TypeScript types
│ ├── __tests__/ # 40+ Vitest test suites │ │ └── api.ts # API response types
│ ├── __tests__/ # 46 Vitest test suites, 830+ tests
│ └── frontend/ │ └── frontend/
│ ├── src/ │ ├── src/
│ │ ├── App.tsx # React router root │ │ ├── App.tsx # React router root
│ │ ├── main.tsx # DOM mount │ │ ├── main.tsx # DOM mount
│ │ ├── api/ # API client functions │ │ ├── api/ # API client functions + TanStack Query hooks
│ │ ├── components/ # Reusable UI components │ │ ├── components/ # Reusable UI (AddUrlModal, MediaServerForm, RatingBadge, StatusBadge, ...)
│ │ ├── contexts/ # React context (auth, etc.) │ │ ├── contexts/ # React context (auth, download progress, scan state)
│ │ ├── hooks/ # Custom hooks (useChannels, etc.) │ │ ├── hooks/ # Custom hooks (useTimezone, useTheme, useChannels, ...)
│ │ ├── pages/ # Page-level components │ │ ├── pages/ # Page-level components (Library, ChannelDetail, Settings, System, ...)
│ │ ├── styles/ # CSS │ │ ├── styles/ # CSS
│ │ └── utils/ # Frontend utilities │ │ └── utils/ # Frontend utilities (format.ts for timezone-aware formatting)
│ ├── index.html # SPA template │ ├── index.html # SPA template (inline theme script for flash prevention)
│ ├── vite.config.ts # Vite build config │ ├── vite.config.ts # Vite build config
│ └── tsconfig.json # Frontend TS config │ └── tsconfig.json # Frontend TS config
├── drizzle/ # Generated migration SQL files ├── drizzle/ # Generated migration SQL files (0001–0017)
├── Dockerfile # Multi-stage Docker build ├── Dockerfile # Multi-stage Docker build
├── docker-compose.yml # Compose orchestration ├── docker-compose.yml # Compose orchestration
├── package.json # Dependencies + scripts ├── package.json # Dependencies + scripts
@ -164,7 +183,19 @@ The download queue runs in-process rather than through Redis/RabbitMQ. Queue sta
Rather than reimplementing platform-specific download logic, Tubearr delegates to yt-dlp via child_process. This provides support for 1000+ platforms, SponsorBlock, subtitle handling, and format selection without maintaining any of that code. The tradeoff is a Python runtime dependency in the Docker image. Rather than reimplementing platform-specific download logic, Tubearr delegates to yt-dlp via child_process. This provides support for 1000+ platforms, SponsorBlock, subtitle handling, and format selection without maintaining any of that code. The tradeoff is a Python runtime dependency in the Docker image.
### Same-origin auth bypass ### Same-origin auth bypass
Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key. Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key. Public paths (`/ping`, `/api/v1/feed/rss`, `/api/v1/media/:id/:filename`) bypass auth entirely.
### Best-effort sidecar pattern
Post-download auxiliary operations (NFO generation, media server scan triggers) are wrapped in try/catch so they never fail the primary download. The download is the primary value; sidecars are additive. Failures are logged for debugging but don't propagate.
### Fire-and-forget media server scans
On download:complete, scans are triggered across all enabled media servers via Promise.allSettled. One failing server doesn't block others or the download pipeline.
### Dual persistence for visual settings
Theme is stored in both localStorage (read by an inline script before React hydrates) and the database (survives server restarts). This prevents flash-of-wrong-theme on page load.
### AbortController for download pause/resume
Pause aborts the in-flight yt-dlp subprocess without incrementing the attempt counter. Resume resets to pending and triggers processNext. This means pause/resume is free from the retry budget perspective.
### Vite dev middleware in Fastify ### Vite dev middleware in Fastify
In development, Vite's middleware is injected into the Fastify server so both backend and frontend run on port 8989 with HMR. In production, the pre-built SPA is served as static files. This avoids CORS issues and simplifies the dev experience. In development, Vite's middleware is injected into the Fastify server so both backend and frontend run on port 8989 with HMR. In production, the pre-built SPA is served as static files. This avoids CORS issues and simplifies the dev experience.
@ -175,14 +206,18 @@ Tubearr has **no external service dependencies** beyond:
- **yt-dlp** — installed via pip in the Docker image (system dependency) - **yt-dlp** — installed via pip in the Docker image (system dependency)
- **ffmpeg** — installed via apk in the Docker image (system dependency) - **ffmpeg** — installed via apk in the Docker image (system dependency)
Optional integrations:
- **Plex** — auto-scan library on download completion
- **Jellyfin** — auto-scan library on download completion
No PostgreSQL, Redis, or other containers required. The Docker image is fully self-contained. No PostgreSQL, Redis, or other containers required. The Docker image is fully self-contained.
## Request Lifecycle ## Request Lifecycle
### API Request ### API Request
``` ```
Browser/Client → nginx01 (TLS) → Fastify (8989) Browser/Client → nginx01 (TLS) → Fastify (8097)
→ Auth middleware (same-origin check OR API key validation) → Auth middleware (same-origin check OR API key validation OR public path bypass)
→ Route handler (validate input, call service/repository) → Route handler (validate input, call service/repository)
→ Drizzle ORM → SQLite (WAL mode) → Drizzle ORM → SQLite (WAL mode)
→ JSON response → JSON response
@ -196,14 +231,27 @@ User clicks "Download" → POST /api/v1/download/:id
→ RateLimiter.acquire(platform) → wait if throttled → RateLimiter.acquire(platform) → wait if throttled
→ DownloadService.downloadItem() → DownloadService.downloadItem()
→ Build yt-dlp args from format profile → Build yt-dlp args from format profile
→ Spawn yt-dlp subprocess → Spawn yt-dlp subprocess (cancellable via AbortController)
→ ProgressParser streams JSON → EventBus → WebSocket → Browser → ProgressParser streams JSON → EventBus → WebSocket → Browser
→ FileOrganizer.organize() → move to /media/Platform/Channel/ → FileOrganizer.organize() → resolve output template → move to /media/...
→ NfoGenerator.writeNfoFile() → Kodi XML sidecar (best-effort)
→ QualityAnalyzer.analyze() → inspect codec/resolution → QualityAnalyzer.analyze() → inspect codec/resolution
→ Update content_items (filePath, fileSize, status=downloaded) → Update content_items (filePath, fileSize, status=downloaded)
→ EventBus.emit('download:complete') → MediaServerService.scanAll() (fire-and-forget)
→ NotificationService.dispatch(onDownload) → NotificationService.dispatch(onDownload)
``` ```
### Ad-hoc URL Download Flow
```
User pastes URL → POST /api/v1/download/url/preview
→ yt-dlp --dump-json --no-download --no-playlist
→ Return metadata preview (title, thumbnail, duration, platform)
User confirms → POST /api/v1/download/url/confirm
→ Create content_item (channelId=null)
→ Enqueue for download
→ Same download flow as above
```
### Channel Monitoring ### Channel Monitoring
``` ```
SchedulerService starts cron job per channel (e.g., every 6 hours) SchedulerService starts cron job per channel (e.g., every 6 hours)
@ -211,6 +259,7 @@ SchedulerService starts cron job per channel (e.g., every 6 hours)
→ yt-dlp --flat-playlist --dump-single-json → yt-dlp --flat-playlist --dump-single-json
→ Parse response → list of content items → Parse response → list of content items
→ Deduplicate against existing platformContentIds in DB → Deduplicate against existing platformContentIds in DB
→ KeywordFilter.matchesKeywordFilter() → apply include/exclude patterns
→ Insert new content_items (status=monitored) → Insert new content_items (status=monitored)
→ If monitoringMode includes auto-download → enqueue items → If monitoringMode includes auto-download → enqueue items
→ Update lastCheckedAt, lastCheckStatus → Update lastCheckedAt, lastCheckStatus
@ -225,8 +274,9 @@ SchedulerService starts cron job per channel (e.g., every 6 hours)
5. Check/update yt-dlp version (production only) 5. Check/update yt-dlp version (production only)
6. Create Vite dev server (development only) 6. Create Vite dev server (development only)
7. Build Fastify server — register plugins, middleware, routes 7. Build Fastify server — register plugins, middleware, routes
8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService 8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService → MediaServerService → MissingFileScanner → NfoGenerator
9. Register platform sources (YouTube, SoundCloud, Generic) 9. Register platform sources (YouTube, SoundCloud, Generic)
10. Start Fastify on `TUBEARR_PORT` 10. Wire EventBus download:complete → media server auto-scan
11. Recover interrupted queue items → restart processing 11. Start Fastify on `TUBEARR_PORT`
12. Start scheduler (monitor enabled channels) 12. Recover interrupted queue items → restart processing
13. Start scheduler (monitor enabled channels)