diff --git a/Architecture.md b/Architecture.md index e481bde..3f6dcec 100644 --- a/Architecture.md +++ b/Architecture.md @@ -28,28 +28,38 @@ graph TB FileSystem["Media Files (/media)"] Notify["Notification Service"] Discord["Discord Webhook"] + MediaServer["Media Servers (Plex/Jellyfin)"] + NfoGen["NFO Generator"] + MissingScan["Missing File Scanner"] + KeywordFilter["Keyword Filter"] + RSS["RSS Feed"] Browser -->|HTTP/WS| Fastify Fastify --> Auth --> Routes Fastify --> WS Routes --> Queue Routes --> Scheduler + Routes --> RSS Scheduler -->|"cron jobs"| Queue + Scheduler -->|"filter"| KeywordFilter Queue -->|"process"| Download Download --> YtDlp YtDlp --> FileSystem Download --> DB + Download -->|"sidecar"| NfoGen --> FileSystem Queue --> Notify --> Discord EventBus -->|"progress"| WS -->|"push"| Browser Download -->|"emit"| EventBus + EventBus -->|"download:complete"| MediaServer Routes --> DB + Routes --> MissingScan --> DB Scheduler --> DB ``` ## Tech Stack | Layer | Technology | Purpose | -|-------|-----------|---------| +|-------|-----------|--------| | HTTP Server | Fastify 5.2.1 | High-performance, plugin-based web framework | | Frontend | React 19 + Vite | SPA with HMR in development | | Router | React Router 7 | Client-side page navigation | @@ -61,7 +71,7 @@ graph TB | Transcoding | ffmpeg | Post-processing, format conversion | | Scheduler | croner | Cron-based channel monitoring | | Icons | Lucide React | UI icon library | -| Testing | Vitest | Unit and integration testing | +| Testing | Vitest | Unit and integration testing (830+ tests) | | TypeScript | 5.7.3 | Type safety across full stack | ## Directory Structure @@ -76,72 +86,81 @@ tubearr/ │ │ ├── index.ts # Database init (WAL mode, foreign keys) │ │ ├── migrate.ts # Migration runner │ │ ├── schema/ # Drizzle table definitions -│ │ │ ├── channels.ts # Monitored channels -│ │ │ ├── content.ts # Content items + format profiles -│ │ │ ├── queue.ts # Download queue +│ │ │ ├── channels.ts # Monitored channels (+ keyword filter columns) +│ │ │ ├── content.ts # Content items + format profiles (+ outputTemplate, contentRating) +│ │ │ ├── queue.ts # Download queue (+ paused status) │ │ │ ├── history.ts # Activity log │ │ │ ├── notifications.ts # Notification settings │ │ │ ├── platform-settings.ts # Per-platform defaults │ │ │ ├── playlists.ts # Playlists + junction table +│ │ │ ├── media-servers.ts # Plex/Jellyfin server configs │ │ │ └── system.ts # Key-value config store │ │ └── repositories/ # Data access layer (one per table) │ ├── server/ │ │ ├── index.ts # Fastify builder, plugin registration │ │ ├── middleware/ -│ │ │ ├── auth.ts # API key + same-origin auth +│ │ │ ├── auth.ts # API key + same-origin auth (public paths: /ping, /feed/rss, /media) │ │ │ └── error-handler.ts # Centralized error responses │ │ └── routes/ # One file per API resource -│ │ ├── channel.ts # Channel CRUD + scan -│ │ ├── content.ts # Content listing + filters +│ │ ├── channel.ts # Channel CRUD + keyword filter updates +│ │ ├── content.ts # Content listing + filters + rating updates + content type counts │ │ ├── download.ts # Enqueue downloads -│ │ ├── queue.ts # Queue management -│ │ ├── format-profile.ts # Quality presets +│ │ ├── adhoc-download.ts # Ad-hoc URL preview + confirm endpoints +│ │ ├── queue.ts # Queue management + pause/resume +│ │ ├── format-profile.ts # Quality presets + output templates │ │ ├── notification.ts # Notification config │ │ ├── platform-settings.ts # Platform defaults │ │ ├── history.ts # Activity log │ │ ├── health.ts # GET /ping -│ │ ├── scan.ts # Manual channel scan +│ │ ├── scan.ts # Manual channel scan (fire-and-forget + scan-all) │ │ ├── collect.ts # Back-catalog import -│ │ ├── system.ts # API key, settings, yt-dlp +│ │ ├── system.ts # Settings, yt-dlp, missing file scan triggers +│ │ ├── media-server.ts # Media server CRUD + test + sections +│ │ ├── feed.ts # RSS 2.0 podcast feed + media streaming │ │ ├── playlist.ts # Playlist data │ │ └── websocket.ts # WebSocket progress stream │ ├── services/ # Business logic layer -│ │ ├── scheduler.ts # Cron-based channel monitoring -│ │ ├── queue.ts # Download queue + concurrency control -│ │ ├── download.ts # Download orchestration (yt-dlp) +│ │ ├── scheduler.ts # Cron-based channel monitoring + keyword filter integration +│ │ ├── queue.ts # Download queue + concurrency + pause/resume via AbortController +│ │ ├── download.ts # Download orchestration (yt-dlp) + NFO sidecar generation │ │ ├── notification.ts # Notification dispatch -│ │ ├── event-bus.ts # Pub/sub for real-time events +│ │ ├── event-bus.ts # Pub/sub for real-time events + download:complete for media servers │ │ ├── progress-parser.ts # Parse yt-dlp JSON progress -│ │ ├── file-organizer.ts # Organize files by platform/channel +│ │ ├── file-organizer.ts # Organize files + output path template resolution +│ │ ├── keyword-filter.ts # Per-channel include/exclude pattern matching (plain, glob, regex) +│ │ ├── media-server.ts # Stateless Plex/Jellyfin scan, test, sections operations +│ │ ├── nfo-generator.ts # Kodi-compatible NFO XML sidecar generation +│ │ ├── missing-file-scanner.ts # Cursor-based batch filesystem existence checking │ │ ├── quality-analyzer.ts # Post-download codec/resolution check │ │ ├── rate-limiter.ts # Per-platform request throttling │ │ ├── cookie-manager.ts # Platform cookie persistence │ │ ├── health.ts # System health diagnostics │ │ └── back-catalog-import.ts # Bulk media import │ ├── sources/ # Platform adapters -│ │ ├── platform-source.ts # Abstract interface -│ │ ├── youtube.ts # YouTube channel/content fetcher +│ │ ├── platform-source.ts # Abstract interface + URL detection (YouTube, SoundCloud, Generic) +│ │ ├── youtube.ts # YouTube channel/content/playlist fetcher │ │ ├── soundcloud.ts # SoundCloud artist/track fetcher -│ │ ├── generic.ts # Fallback URL handler +│ │ ├── generic.ts # Fallback URL handler (any yt-dlp-supported site) │ │ └── yt-dlp.ts # yt-dlp subprocess wrapper │ ├── types/ -│ │ └── index.ts # Shared TypeScript types -│ ├── __tests__/ # 40+ Vitest test suites +│ │ ├── index.ts # Shared TypeScript types +│ │ └── api.ts # API response types +│ ├── __tests__/ # 46 Vitest test suites, 830+ tests │ └── frontend/ │ ├── src/ │ │ ├── App.tsx # React router root │ │ ├── main.tsx # DOM mount -│ │ ├── api/ # API client functions -│ │ ├── components/ # Reusable UI components -│ │ ├── contexts/ # React context (auth, etc.) -│ │ ├── hooks/ # Custom hooks (useChannels, etc.) -│ │ ├── pages/ # Page-level components +│ │ ├── api/ # API client functions + TanStack Query hooks +│ │ ├── components/ # Reusable UI (AddUrlModal, MediaServerForm, RatingBadge, StatusBadge, ...) +│ │ ├── contexts/ # React context (auth, download progress, scan state) +│ │ ├── hooks/ # Custom hooks (useTimezone, useTheme, useChannels, ...) +│ │ ├── pages/ # Page-level components (Library, ChannelDetail, Settings, System, ...) │ │ ├── styles/ # CSS -│ │ └── utils/ # Frontend utilities -│ ├── index.html # SPA template +│ │ └── utils/ # Frontend utilities (format.ts for timezone-aware formatting) +│ ├── index.html # SPA template (inline theme script for flash prevention) │ ├── vite.config.ts # Vite build config │ └── tsconfig.json # Frontend TS config -├── drizzle/ # Generated migration SQL files +├── drizzle/ # Generated migration SQL files (0001–0017) ├── Dockerfile # Multi-stage Docker build ├── docker-compose.yml # Compose orchestration ├── package.json # Dependencies + scripts @@ -164,7 +183,19 @@ The download queue runs in-process rather than through Redis/RabbitMQ. Queue sta Rather than reimplementing platform-specific download logic, Tubearr delegates to yt-dlp via child_process. This provides support for 1000+ platforms, SponsorBlock, subtitle handling, and format selection without maintaining any of that code. The tradeoff is a Python runtime dependency in the Docker image. ### Same-origin auth bypass -Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key. +Browser requests from the Tubearr UI skip API key authentication. This means the React SPA doesn't need to store or transmit an API key — it's authenticated implicitly by running on the same origin. External clients (scripts, other *arr apps) use the API key. Public paths (`/ping`, `/api/v1/feed/rss`, `/api/v1/media/:id/:filename`) bypass auth entirely. + +### Best-effort sidecar pattern +Post-download auxiliary operations (NFO generation, media server scan triggers) are wrapped in try/catch so they never fail the primary download. The download is the primary value; sidecars are additive. Failures are logged for debugging but don't propagate. + +### Fire-and-forget media server scans +On download:complete, scans are triggered across all enabled media servers via Promise.allSettled. One failing server doesn't block others or the download pipeline. + +### Dual persistence for visual settings +Theme is stored in both localStorage (read by an inline script before React hydrates) and the database (survives server restarts). This prevents flash-of-wrong-theme on page load. + +### AbortController for download pause/resume +Pause aborts the in-flight yt-dlp subprocess without incrementing the attempt counter. Resume resets to pending and triggers processNext. This means pause/resume is free from the retry budget perspective. ### Vite dev middleware in Fastify In development, Vite's middleware is injected into the Fastify server so both backend and frontend run on port 8989 with HMR. In production, the pre-built SPA is served as static files. This avoids CORS issues and simplifies the dev experience. @@ -175,14 +206,18 @@ Tubearr has **no external service dependencies** beyond: - **yt-dlp** — installed via pip in the Docker image (system dependency) - **ffmpeg** — installed via apk in the Docker image (system dependency) +Optional integrations: +- **Plex** — auto-scan library on download completion +- **Jellyfin** — auto-scan library on download completion + No PostgreSQL, Redis, or other containers required. The Docker image is fully self-contained. ## Request Lifecycle ### API Request ``` -Browser/Client → nginx01 (TLS) → Fastify (8989) - → Auth middleware (same-origin check OR API key validation) +Browser/Client → nginx01 (TLS) → Fastify (8097) + → Auth middleware (same-origin check OR API key validation OR public path bypass) → Route handler (validate input, call service/repository) → Drizzle ORM → SQLite (WAL mode) → JSON response @@ -196,14 +231,27 @@ User clicks "Download" → POST /api/v1/download/:id → RateLimiter.acquire(platform) → wait if throttled → DownloadService.downloadItem() → Build yt-dlp args from format profile - → Spawn yt-dlp subprocess + → Spawn yt-dlp subprocess (cancellable via AbortController) → ProgressParser streams JSON → EventBus → WebSocket → Browser - → FileOrganizer.organize() → move to /media/Platform/Channel/ + → FileOrganizer.organize() → resolve output template → move to /media/... + → NfoGenerator.writeNfoFile() → Kodi XML sidecar (best-effort) → QualityAnalyzer.analyze() → inspect codec/resolution → Update content_items (filePath, fileSize, status=downloaded) + → EventBus.emit('download:complete') → MediaServerService.scanAll() (fire-and-forget) → NotificationService.dispatch(onDownload) ``` +### Ad-hoc URL Download Flow +``` +User pastes URL → POST /api/v1/download/url/preview + → yt-dlp --dump-json --no-download --no-playlist + → Return metadata preview (title, thumbnail, duration, platform) +User confirms → POST /api/v1/download/url/confirm + → Create content_item (channelId=null) + → Enqueue for download + → Same download flow as above +``` + ### Channel Monitoring ``` SchedulerService starts cron job per channel (e.g., every 6 hours) @@ -211,6 +259,7 @@ SchedulerService starts cron job per channel (e.g., every 6 hours) → yt-dlp --flat-playlist --dump-single-json → Parse response → list of content items → Deduplicate against existing platformContentIds in DB + → KeywordFilter.matchesKeywordFilter() → apply include/exclude patterns → Insert new content_items (status=monitored) → If monitoringMode includes auto-download → enqueue items → Update lastCheckedAt, lastCheckStatus @@ -225,8 +274,9 @@ SchedulerService starts cron job per channel (e.g., every 6 hours) 5. Check/update yt-dlp version (production only) 6. Create Vite dev server (development only) 7. Build Fastify server — register plugins, middleware, routes -8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService +8. Initialize services in dependency order: RateLimiter → FileOrganizer → CookieManager → QualityAnalyzer → DownloadService → EventBus → QueueService → SchedulerService → NotificationService → HealthService → MediaServerService → MissingFileScanner → NfoGenerator 9. Register platform sources (YouTube, SoundCloud, Generic) -10. Start Fastify on `TUBEARR_PORT` -11. Recover interrupted queue items → restart processing -12. Start scheduler (monitor enabled channels) \ No newline at end of file +10. Wire EventBus download:complete → media server auto-scan +11. Start Fastify on `TUBEARR_PORT` +12. Recover interrupted queue items → restart processing +13. Start scheduler (monitor enabled channels) \ No newline at end of file