# Chrysopedia

**Knowledge base for music production techniques**, extracted from video content via an LLM-powered pipeline. Producers can search for specific techniques and find timestamped key moments, signal chains, and synthesized study guides — all derived from creator videos.

## Current State

Fifteen milestones complete. The system is deployed and running on ub01 at `http://ub01:8096`.

### What's Built

- **Transcription pipeline** — Desktop script: video → ffmpeg audio extraction → Whisper large-v3 → timestamped transcript JSON
- **6-stage LLM extraction pipeline** — Transcript segmentation → key moment extraction → classification/tagging → technique page synthesis → embedding/indexing. Per-stage model routing (chat vs thinking models). Resumable per-video per-stage.
- **Article versioning** — Pre-overwrite snapshots with pipeline metadata (prompt SHA-256 hashes, model config) for benchmarking extraction quality across prompt iterations.
- **Review queue UI** — Admin interface for approving/editing/rejecting extracted key moments.
- **Search-first web UI** — Dark theme with cyan accents. Semantic search (Qdrant), topic/creator browse pages, technique detail pages with signal chain blocks and video source metadata. Search autocomplete with popular suggestions on focus and typeahead on 2+ characters.
- **Docker Compose deployment** — API, worker, web, watcher, PostgreSQL, Redis, Qdrant, Ollama on ub01 with bind mounts at `/vmPool/r/services/chrysopedia_*`.
- **Transcript folder watcher** — Standalone service monitors `/vmPool/r/services/chrysopedia_watch/` for new transcript JSON files, validates structure, POSTs to ingest API. Processed files move to `processed/`, failures to `failed/` with `.error` sidecar. Uses watchdog PollingObserver for ZFS reliability.
- **Prompt template system** — Editable per-stage prompt files with SHA-256 tracking for reproducibility.
- **Pipeline admin dashboard** — Admin page at `/admin/pipeline` with video list, status badges, retrigger/revoke controls, expandable event log with token usage, collapsible JSON response viewer, and live worker status indicator. Pipeline events instrumented via `PipelineEvent` model (stage, event_type, token counts, JSONB payload).
- **Technique page 2-column layout** — Prose content (summary + study guide) on left, sidebar (key moments, signal chains, plugins, related techniques) on right at desktop widths. Sticky sidebar. Collapses to single column at ≤768px.
- **Key moment card redesign** — Cards in sidebar show title prominently on its own line (h3), with source file, timestamp, and content type badge on a clean metadata row below.
- **Admin navigation dropdown** — Header consolidated: Home, Topics, Creators visible; Admin dropdown reveals Review, Reports, Pipeline. ModeToggle removed from header.
- **Pipeline head/tail log view** — Pipeline event log shows Head/Tail toggle with configurable event count. Token counts visible per event and per video.
- **Pipeline debug mode** — Redis-backed toggle captures full LLM system prompt, user prompt, and response text in pipeline_events. Per-stage token summary endpoint for cost/usage analysis.
- **Debug payload viewer** — Admin UI component for viewing, copying, and exporting full LLM I/O from debug-mode pipeline runs.
- **Admin UX improvements** — Debug mode toggle, status filter pills, cleaner labels, pruned dead UI elements, review queue cross-links.
- **Git commit SHA tracking** — Pipeline captures current git commit SHA. Technique page versions display commit hash in metadata.
- **Technique page tag polish** — Sidebar reordered (Plugins Referenced at top), creator name prominent, tags use coherent color system.
- **Topics page redesign** — 7 categories (including Music Theory) with card layout, descriptions, sub-topic counts, colored left borders.
- **App footer** — Every page shows version, build date, commit SHA (linked to GitHub), and repo link. Build-time constants via Vite define with Docker ARG/ENV passthrough.
- **Key moment search links fixed** — Search results for key moments link to parent technique page with anchor scroll to the specific moment.
- **Test data and jargon cleanup** — Removed TestCreator from production data. Removed yellow "livestream sourced" jargon banner from search results. Clean footer version info.
- **Homepage card enrichment** — Technique cards on homepage show topic tag pills and key moment count. Creator detail pages show technique-count-by-topic instead of meaningless '0 views'.
- **Homepage first impression** — Hero tagline ("Production Knowledge, Distilled"), value proposition, 3-step how-it-works grid, "Start Exploring" CTA, popular topic quick-links, featured technique spotlight (random), and enriched 2-column recently-added grid with deduplication.
- **About page** — /about with what/how/who sections (purpose, 5-step extraction pipeline, maintainer links). Footer link added.
- **Dedicated sub-topic pages** — Clicking a sub-topic on Topics page loads `/topics/:category/:subtopic` with techniques grouped by creator, breadcrumb navigation, and loading/error/empty states.
- **Related techniques cross-linking** — Every technique page shows up to 4 related techniques scored by creator overlap, topic match, and shared tags. Curated links take priority; dynamic results fill remaining slots.
- **Topic color coding & page transitions** — Per-category accent colors on sub-topic pages and search results. CSS-only fade-in page enter animation on all 7 public pages.
- **Search autocomplete** — Shared SearchAutocomplete component with popular suggestions on empty focus and debounced typeahead on 2+ characters. Used on Home and SearchResults pages.
- **Interaction delight** — Card hover animations (scale + shadow), staggered entrance animations on card grids, featured technique glow treatment, Random Technique discovery button.
- **Topics collapse/expand** — Topics page loads with all categories collapsed. Smooth CSS grid-template-rows animation on expand/collapse.
- **Creator stats pills** — Creator detail page topic stats rendered as colored pill badges matching category colors.
- **Tag overflow** — Shared TagList component caps visible tags at 4 with "+N more" overflow pill. Applied across all 5 tag-rendering sites.
- **Empty subtopic handling** — Subtopics with 0 techniques show "Coming soon" badge instead of dead-end links.
- **Accessibility & SEO fixes** — Single h1 per page, skip-to-content keyboard link, AA-compliant muted text contrast (#828291), descriptive per-route browser tab titles via useDocumentTitle hook.
- **Multi-field composite search** — Search tokenizes multi-word queries, AND-matches each token across creator/title/tags/category/body fields. Partial matches fallback when no exact cross-field match exists. Qdrant embeddings enriched with creator names and topic tags. Admin reindex-all endpoint for re-embedding after changes.
- **Sort controls on all list views** — Reusable SortDropdown component on SearchResults, SubTopicPage, and CreatorDetail. Sort options: relevance/newest/oldest/alpha/creator (context-appropriate per page). Preference persists in sessionStorage across navigation.
- **Prompt quality toolkit** — CLI tool (`python -m pipeline.quality`) with: LLM fitness suite (9 tests across Mandelbrot reasoning, JSON compliance, instruction following, diverse battery), 5-dimension quality scorer with voice preservation dial (3-band prompt modification), automated prompt A/B optimization loop (LLM-powered variant generation, iterative scoring, leaderboard/trajectory reporting), multi-stage support for pipeline stages 2-5 with per-stage rubrics and fixtures.
- **Search query logging** — All non-empty searches logged to PostgreSQL search_log table via fire-and-forget async pattern. GET /api/v1/search/popular returns top 10 queries from last 7 days with Redis read-through cache (5-min TTL).
- **Social proof & freshness signals** — Homepage stats scorecard (article count, creator count), trending searches pill section, and date stamps on recently-added cards. Creators browse page shows "Last updated" per creator. Admin dropdown opens on hover at desktop widths.
- **Multi-source technique pages** — Technique pages restructured to support multiple source videos per page. Nested H2/H3 body sections with table of contents and inline [N] citation markers linking prose claims to source key moments. Composition pipeline merges new video moments into existing pages with offset-based citation re-indexing and deduplication. Format-discriminated rendering (v1 dict / v2 list-of-objects) preserves backward compatibility. Per-section Qdrant embeddings with deterministic UUIDs enable section-level search results with deep-link scrolling. Admin view at /admin/techniques for multi-source page management.

### Stack

- **Backend:** FastAPI + Celery + SQLAlchemy (async) + PostgreSQL + Redis + Qdrant
- **Frontend:** React + TypeScript + Vite
- **LLM:** OpenAI-compatible API (DGX Sparks Qwen primary, local Ollama fallback)
- **Deployment:** Docker Compose on ub01, nginx reverse proxy on nuc01

### Milestone History

| ID | Title | Status |
|----|-------|--------|
| M001 | Chrysopedia Foundation — Infrastructure, Pipeline Core, and Skeleton UI | ✅ Complete |
| M002 | Deployment — GitHub, ub01 Docker Stack, and Production Wiring | ✅ Complete |
| M003 | Domain + DNS + Per-Stage LLM Model Routing | ✅ Complete |
| M004 | UI Polish, Bug Fixes, Technique Page Redesign, and Article Versioning | ✅ Complete |
| M005 | Pipeline Dashboard, Technique Page Redesign, Key Moment Cards | ✅ Complete |
| M006 | Admin Nav, Pipeline Log Views, Commit SHA, Tag Polish, Topics Redesign, Footer | ✅ Complete |
| M007 | Pipeline Transparency, Auto-Ingest, Admin UX Polish, and Mobile Fixes | ✅ Complete |
| M008 | Credibility Debt Cleanup — Broken Links, Test Data, Jargon, Empty Metrics | ✅ Complete |
| M009 | Homepage & First Impression | ✅ Complete |
| M010 | Discovery, Navigation & Visual Identity | ✅ Complete |
| M011 | Interaction Polish, Navigation & Accessibility | ✅ Complete |
| M012 | Multi-Field Composite Search & Sort Controls | ✅ Complete |
| M013 | Prompt Quality Toolkit — LLM Fitness, Scoring, and Automated Optimization | ✅ Complete |
| M014 | Multi-Source Technique Pages — Nested Sections, Composition, Citations, and Section Search | ✅ Complete |
| M015 | Social Proof, Freshness Signals & Admin UX | ✅ Complete |
