docs: add Newcomer Guide — full platform walkthrough with 8 sections
jlightner
parent
ba72230478
commit
0cde553f75
1 changed files with 198 additions and 0 deletions
198
Newcomer-Guide.md
Normal file
198
Newcomer-Guide.md
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
# Newcomer Guide
|
||||
|
||||
Welcome to Chrysopedia — this guide takes you from zero to productive with the entire platform.
|
||||
|
||||
## What is Chrysopedia
|
||||
|
||||
Chrysopedia is an AI-powered knowledge base for music production techniques, built from video content by creators. It extracts, structures, and makes searchable the knowledge embedded in production tutorials and livestreams. The result is a browsable, searchable library of technique pages, key moments, and creator-attributed study guides — all derived automatically from video transcripts.
|
||||
|
||||
## How Content Flows
|
||||
|
||||
Content enters Chrysopedia as video transcripts and exits as structured, searchable knowledge:
|
||||
|
||||
1. **Video transcripts** (Whisper large-v3 on RTX 4090) land as JSON files in a watched folder
|
||||
2. The **6-stage pipeline** processes them automatically:
|
||||
- **Stage 1 — Transcript Segmentation:** Splits raw transcripts into coherent segments
|
||||
- **Stage 2 — Key Moment Extraction:** Identifies technique demonstrations, tips, and notable passages
|
||||
- **Stage 3 — Classification & Tagging:** Assigns topic categories and tags to each key moment (7 top-level categories: Sound Design, Mixing, Arrangement, Sampling, Music Theory, Workflow, Sound Selection)
|
||||
- **Stage 4 — Technique Page Synthesis:** Generates study-guide prose with v2 structured body sections, signal chains, and citations referencing source key moments
|
||||
- **Stage 5 — Embedding & Indexing:** Embeds technique pages and key moments into Qdrant vector store and LightRAG knowledge graph for semantic search
|
||||
- **Stage 6 — Highlight Detection:** (Optional) Scores key moments across 10 dimensions for editorial curation
|
||||
3. **Results** appear as structured technique pages, searchable key moments, and cross-references between creators
|
||||
|
||||
Each stage is a Celery task. The pipeline orchestrator chains them and tracks status per video. See [[Pipeline]] for stage-level detail and prompt templates.
|
||||
|
||||
## Using the Web UI
|
||||
|
||||
### Search
|
||||
|
||||
- **Homepage search bar** or **Cmd+K** from any page opens the search interface
|
||||
- Combines **semantic search** (LightRAG knowledge graph + Qdrant vectors) with **keyword search** (PostgreSQL full-text)
|
||||
- Results show technique pages and key moments with creator attribution and relevance scores
|
||||
- Multi-token queries use AND logic with partial-match fallback — searching "keota snare" finds content where "keota" matches the creator and "snare" matches technique content
|
||||
|
||||
### Browse Topics
|
||||
|
||||
- 7 top-level categories → sub-topics → technique pages grouped by creator
|
||||
- Categories expand/collapse with CSS grid animation
|
||||
- Each technique page card shows creator, topic tags, and key moment count
|
||||
|
||||
### Browse Creators
|
||||
|
||||
- Filterable list of all creators with genre tags
|
||||
- Randomized default sort (no alphabetical bias)
|
||||
- Click through to creator detail pages showing all their technique pages and key moments
|
||||
|
||||
### Technique Pages
|
||||
|
||||
- **Study guide prose** — LLM-synthesized content with structured sections (v2 body format)
|
||||
- **Key moments index** — timestamped references back to source videos with citation markers
|
||||
- **Related techniques** — cross-references to similar content from other creators
|
||||
- **Table of contents** — auto-generated from section headings, displayed in sidebar
|
||||
- **Reading header** — sticky section indicator bar that appears when scrolling past the page title
|
||||
- **Inline player** — audio/video player with chapter markers and key moment timeline pins
|
||||
|
||||
### Chat
|
||||
|
||||
- **Creator-scoped AI chat** — ask questions about any creator's techniques
|
||||
- **Citation support** — responses include numbered source references
|
||||
- **Cascade retrieval** — queries search creator-specific context first, then domain (topic category), then global knowledge
|
||||
- **Multi-turn memory** — conversation context persists within a session
|
||||
- **Streaming responses** — SSE-based token streaming with source metadata sent first
|
||||
- **Quality toolkit** — refined system prompt with baseline quality metrics
|
||||
|
||||
## For Creators
|
||||
|
||||
### Registration & Onboarding
|
||||
|
||||
- Standard registration with email/password
|
||||
- **3-step onboarding wizard** (shown once after first login):
|
||||
1. Welcome message explaining the platform
|
||||
2. Content consent selection (choose which content types to publish)
|
||||
3. Quick tour of available features
|
||||
- Onboarding completion tracked via `onboarding_completed` flag on user profile
|
||||
|
||||
### Consent Dashboard
|
||||
|
||||
- Control which content types are published (technique pages, key moments, chat availability)
|
||||
- Granular per-content-type toggles
|
||||
- Changes take effect on next pipeline run
|
||||
|
||||
### Creator Dashboard
|
||||
|
||||
- Overview stats: total technique pages, key moments, video count
|
||||
- Recent posts and activity feed
|
||||
- Quick links to technique pages derived from your content
|
||||
|
||||
### Transparency Page
|
||||
|
||||
- View all entities, relationships, and technique pages derived from your content
|
||||
- Expandable/collapsible category sections (CSS grid animation)
|
||||
- Full audit trail of what the system extracted from your videos
|
||||
|
||||
### Data Export
|
||||
|
||||
- **GDPR-style ZIP download** of all derived content via `GET /creator/export`
|
||||
- Includes technique pages, key moments, classifications, and metadata
|
||||
- One-click download from creator dashboard
|
||||
|
||||
### Notifications
|
||||
|
||||
- **Email digest** of platform activity (new technique pages, key moments from your content)
|
||||
- Configurable frequency: daily, weekly, or disabled
|
||||
- **Signed unsubscribe links** (PyJWT tokens) — one-click unsubscribe without login
|
||||
- Managed via notification preferences in creator settings
|
||||
|
||||
### Personality Profiles
|
||||
|
||||
- 5-tier system for chat persona customization based on creator teaching style
|
||||
- LLM-extracted from creator's content patterns
|
||||
- Influences how the chat engine responds to questions about that creator's techniques
|
||||
- See [[Personality-Profiles]] for tier definitions
|
||||
|
||||
## For Admins
|
||||
|
||||
### Review Queue
|
||||
|
||||
- Approve, reject, or edit key moments organized by source video
|
||||
- Bulk actions for efficient moderation
|
||||
- Filter by creator, video, or processing status
|
||||
|
||||
### Pipeline Admin
|
||||
|
||||
- Monitor processing stages for all videos in the system
|
||||
- Filter by creator, status (pending, processing, complete, failed)
|
||||
- View per-stage timing and error details
|
||||
|
||||
### Usage Dashboard
|
||||
|
||||
- **Token consumption tracking** — LLM API usage over time
|
||||
- **Top creators and users** — ranked by content volume and platform usage
|
||||
- **Daily statistics** — requests, chat sessions, search queries
|
||||
- **Rate limiting visibility** — sliding-window rate limiter status (Redis-backed, per-user)
|
||||
|
||||
### Audit Log
|
||||
|
||||
- All administrative actions tracked with timestamp, actor, and details
|
||||
- Searchable and filterable
|
||||
|
||||
### User Management
|
||||
|
||||
- Role assignment (admin, creator, user)
|
||||
- Account status management (active, suspended)
|
||||
- Impersonation support for debugging user-specific issues (see [[Impersonation]])
|
||||
|
||||
## Adding New Content
|
||||
|
||||
1. **Prepare transcripts** — Run Whisper large-v3 on video/audio files (RTX 4090 recommended for speed). Output format: JSON with timestamps.
|
||||
2. **Place in watched folder** — Drop transcript JSON files into the configured watch directory. The folder watcher (PollingObserver, works on ZFS/NFS) detects new files automatically.
|
||||
3. **Pipeline processes automatically** — All 6 stages run in sequence via Celery task chain. Monitor progress in the Pipeline Admin panel.
|
||||
4. **File stability check** — The watcher waits for file size to stabilize (2-second check) before processing, handling partial SCP/rsync writes safely.
|
||||
5. **Quality control** — New key moments appear in the Review Queue for admin approval before publishing.
|
||||
6. **Re-processing** — To update existing content, re-drop the transcript. The pipeline handles upserts (though Qdrant point deduplication is a known improvement area).
|
||||
|
||||
## Infrastructure & Deployment
|
||||
|
||||
Chrysopedia runs as a **Docker Compose stack on ub01**:
|
||||
|
||||
| Service | Purpose |
|
||||
|---------|---------|
|
||||
| `chrysopedia-api` | FastAPI application server |
|
||||
| `chrysopedia-worker` | Celery worker + Beat scheduler (email digests, periodic tasks) |
|
||||
| `chrysopedia-web-8096` | nginx reverse proxy serving frontend + API routing |
|
||||
| `chrysopedia-db` | PostgreSQL 16 (port 5433 externally) |
|
||||
| `chrysopedia-redis` | Redis — caching, rate limiting, Celery broker, classification data |
|
||||
| `chrysopedia-qdrant` | Qdrant vector database for semantic search |
|
||||
| `chrysopedia-ollama` | Ollama — local LLM fallback when primary DGX endpoint is unavailable |
|
||||
| `chrysopedia-lightrag` | LightRAG knowledge graph for entity-aware retrieval |
|
||||
|
||||
- **Primary LLM:** DGX endpoint with **automatic Ollama fallback** (fail-open, configurable via `LLM_FALLBACK_URL` / `LLM_FALLBACK_MODEL`)
|
||||
- **Web UI:** `http://ub01:8096`
|
||||
- **External:** `https://chrysopedia.com` via nuc01 nginx reverse proxy
|
||||
|
||||
For full deployment instructions and rebuild commands, see [[Deployment]].
|
||||
For local development setup and common gotchas, see [[Development-Guide]].
|
||||
|
||||
## Where to Learn More
|
||||
|
||||
Every aspect of Chrysopedia is documented in the wiki:
|
||||
|
||||
| Topic | Wiki Page |
|
||||
|-------|-----------|
|
||||
| System architecture, Docker services, network topology | [[Architecture]] |
|
||||
| All 80+ API endpoints grouped by domain | [[API-Surface]] |
|
||||
| SQLAlchemy models, relationships, enums | [[Data-Model]] |
|
||||
| Semantic + keyword search, LightRAG cascade | [[Search-Retrieval]] |
|
||||
| Streaming Q&A, multi-turn memory, fallback | [[Chat-Engine]] |
|
||||
| 6-stage LLM extraction pipeline, prompt system | [[Pipeline]] |
|
||||
| Audio/video player, chapter markers, timeline pins | [[Player]] |
|
||||
| 10-dimension highlight scoring and review | [[Highlights]] |
|
||||
| LLM-extracted creator teaching personality | [[Personality-Profiles]] |
|
||||
| JWT authentication, roles, permissions | [[Authentication]] |
|
||||
| Admin impersonation for debugging | [[Impersonation]] |
|
||||
| Environment variables and feature flags | [[Configuration]] |
|
||||
| Docker Compose setup, rebuild commands | [[Deployment]] |
|
||||
| Prometheus metrics, health checks, logging | [[Monitoring]] |
|
||||
| Local dev setup, common gotchas | [[Development-Guide]] |
|
||||
| Architectural decisions register (D001–D048) | [[Decisions]] |
|
||||
| LLM agent context injection system | [[Agent-Context]] |
|
||||
Loading…
Add table
Reference in a new issue