xpltdco/chrysopedia
2
0
Fork 0
Code Issues 2 Pull requests Projects Releases Packages Wiki Activity Actions

docs: Created comprehensive README.md with architecture diagram, setup…

Browse source 
- "README.md"
- "tests/fixtures/sample_transcript.json"

GSD-Task: S01/T05
This commit is contained in:
jlightner 2026-03-29 22:00:41 +00:00
parent a79282ffa9
commit 9bddaed5d1
2 changed files with 427 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

279
README.md Normal file
View file

@ -0,0 +1,279 @@
# Chrysopedia
> From *chrysopoeia* (alchemical transmutation of base material into gold) + *encyclopedia*.
> Chrysopedia transmutes raw video content into refined, searchable production knowledge.
A self-hosted knowledge extraction and retrieval system for electronic music production content. Transcribes video libraries with Whisper, extracts key moments and techniques with LLM analysis, and serves a search-first web UI for mid-session retrieval.
---
## Architecture
```
┌──────────────────────────────────────────────────────────────────┐
│ Desktop (GPU workstation) │
│ ┌──────────────┐ │
│ │ whisper/ │ Transcribes video → JSON (Whisper large-v3) │
│ │ transcribe.py │ Runs locally with CUDA, outputs to /data │
│ └──────┬───────┘ │
│ │ JSON transcripts │
└─────────┼────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────┐
│ Docker Compose (xpltd_chrysopedia) — Server (e.g. ub01) │
│ │
│ ┌────────────────┐ ┌────────────────┐ ┌──────────────────┐ │
│ │ chrysopedia-db │ │chrysopedia-redis│ │ chrysopedia-api │ │
│ │ PostgreSQL 16 │ │ Redis 7 │ │ FastAPI + Uvicorn│ │
│ │ :5433→5432 │ │ │ │ :8000 │ │
│ └────────────────┘ └────────────────┘ └────────┬─────────┘ │
│ │ │
│ ┌──────────────────┐ ┌──────────────────────┐ │ │
│ │ chrysopedia-web │ │ chrysopedia-worker │ │ │
│ │ React + nginx │ │ Celery (LLM pipeline)│ │ │
│ │ :3000→80 │ │ │ │ │
│ └──────────────────┘ └──────────────────────┘ │ │
│ │ │
│ Network: chrysopedia (172.24.0.0/24) │ │
└──────────────────────────────────────────────────────────────────┘
```
### Services
| Service | Image / Build | Port | Purpose |
|----------------------|------------------------|---------------|--------------------------------------------|
| `chrysopedia-db` | `postgres:16-alpine` | `5433 → 5432` | Primary data store (7 entity schema) |
| `chrysopedia-redis` | `redis:7-alpine` | — | Celery broker / cache |
| `chrysopedia-api` | `docker/Dockerfile.api`| `8000` | FastAPI REST API |
| `chrysopedia-worker` | `docker/Dockerfile.api`| — | Celery worker for LLM pipeline stages 2-5 |
| `chrysopedia-web` | `docker/Dockerfile.web`| `3000 → 80` | React frontend (nginx) |
### Data Model (7 entities)
- **Creator** — artists/producers whose content is indexed
- **SourceVideo** — original video files processed by the pipeline
- **TranscriptSegment** — timestamped text segments from Whisper
- **KeyMoment** — discrete insights extracted by LLM analysis
- **TechniquePage** — synthesized knowledge pages (primary output)
- **RelatedTechniqueLink** — cross-references between technique pages
- **Tag** — hierarchical topic/genre taxonomy
---
## Prerequisites
- **Docker** ≥ 24.0 and **Docker Compose** ≥ 2.20
- **Python 3.10+** (for the Whisper transcription script)
- **ffmpeg** (for audio extraction)
- **NVIDIA GPU + CUDA** (recommended for Whisper; CPU fallback available)
---
## Quick Start
### 1. Clone and configure
```bash
git clone <repository-url>
cd content-to-kb-automator
# Create environment file from template
cp .env.example .env
# Edit .env with your actual values (see Environment Variables below)
```
### 2. Start the Docker Compose stack
```bash
docker compose up -d
```
This starts PostgreSQL, Redis, the API server, the Celery worker, and the web UI.
### 3. Run database migrations
```bash
# From inside the API container:
docker compose exec chrysopedia-api alembic upgrade head
# Or locally (requires Python venv with backend deps):
alembic upgrade head
```
### 4. Verify the stack
```bash
# Health check (with DB connectivity)
curl http://localhost:8000/health
# API health (lightweight, no DB)
curl http://localhost:8000/api/v1/health
# Docker Compose status
docker compose ps
```
### 5. Transcribe videos (desktop)
```bash
cd whisper
pip install -r requirements.txt
# Single file
python transcribe.py --input "path/to/video.mp4" --output-dir ./transcripts
# Batch (all videos in a directory)
python transcribe.py --input ./videos/ --output-dir ./transcripts
```
See [`whisper/README.md`](whisper/README.md) for full transcription documentation.
---
## Environment Variables
Create `.env` from `.env.example`. All variables have sensible defaults for local development.
### Database
| Variable | Default | Description |
|--------------------|----------------|---------------------------------|
| `POSTGRES_USER` | `chrysopedia` | PostgreSQL username |
| `POSTGRES_PASSWORD`| `changeme` | PostgreSQL password |
| `POSTGRES_DB` | `chrysopedia` | Database name |
| `DATABASE_URL` | *(composed)* | Full async connection string |
### Services
| Variable | Default | Description |
|-----------------|------------------------------------|--------------------------|
| `REDIS_URL` | `redis://chrysopedia-redis:6379/0` | Redis connection string |
### LLM Configuration
| Variable | Default | Description |
|---------------------|-------------------------------------------|------------------------------------|
| `LLM_API_URL` | `https://friend-openwebui.example.com/api`| Primary LLM endpoint (OpenAI-compatible) |
| `LLM_API_KEY` | `sk-changeme` | API key for primary LLM |
| `LLM_MODEL` | `qwen2.5-72b` | Primary model name |
| `LLM_FALLBACK_URL` | `http://localhost:11434/v1` | Fallback LLM endpoint (Ollama) |
| `LLM_FALLBACK_MODEL`| `qwen2.5:14b-q8_0` | Fallback model name |
### Embedding / Vector
| Variable | Default | Description |
|-----------------------|-------------------------------|--------------------------|
| `EMBEDDING_API_URL` | `http://localhost:11434/v1` | Embedding endpoint |
| `EMBEDDING_MODEL` | `nomic-embed-text` | Embedding model name |
| `QDRANT_URL` | `http://qdrant:6333` | Qdrant vector DB URL |
| `QDRANT_COLLECTION` | `chrysopedia` | Qdrant collection name |
### Application
| Variable | Default | Description |
|--------------------------|----------------------------------|--------------------------------|
| `APP_ENV` | `production` | Environment (`development` / `production`) |
| `APP_LOG_LEVEL` | `info` | Log level |
| `APP_SECRET_KEY` | `changeme-generate-a-real-secret`| Application secret key |
| `TRANSCRIPT_STORAGE_PATH`| `/data/transcripts` | Transcript JSON storage path |
| `VIDEO_METADATA_PATH` | `/data/video_meta` | Video metadata storage path |
| `REVIEW_MODE` | `true` | Enable human review workflow |
---
## Development Workflow
### Local development (without Docker)
```bash
# Create virtual environment
python -m venv .venv
source .venv/bin/activate
# Install backend dependencies
pip install -r backend/requirements.txt
# Start PostgreSQL and Redis (via Docker)
docker compose up -d chrysopedia-db chrysopedia-redis
# Run migrations
alembic upgrade head
# Start the API server with hot-reload
cd backend && uvicorn main:app --reload --host 0.0.0.0 --port 8000
```
### Database migrations
```bash
# Create a new migration after model changes
alembic revision --autogenerate -m "describe_change"
# Apply all pending migrations
alembic upgrade head
# Rollback one migration
alembic downgrade -1
```
### Project structure
```
content-to-kb-automator/
├── backend/ # FastAPI application
│ ├── main.py # App entry point, middleware, routers
│ ├── config.py # pydantic-settings configuration
│ ├── database.py # SQLAlchemy async engine + session
│ ├── models.py # 7-entity ORM models
│ ├── schemas.py # Pydantic request/response schemas
│ ├── routers/ # API route handlers
│ │ ├── health.py # /health (DB check)
│ │ ├── creators.py # /api/v1/creators
│ │ └── videos.py # /api/v1/videos
│ └── requirements.txt # Python dependencies
├── whisper/ # Desktop transcription script
│ ├── transcribe.py # Whisper CLI tool
│ ├── requirements.txt # Whisper + ffmpeg deps
│ └── README.md # Transcription documentation
├── docker/ # Dockerfiles
│ ├── Dockerfile.api # FastAPI + Celery image
│ ├── Dockerfile.web # React + nginx image
│ └── nginx.conf # nginx reverse proxy config
├── alembic/ # Database migrations
│ ├── env.py # Migration environment
│ └── versions/ # Migration scripts
├── config/ # Configuration files
│ └── canonical_tags.yaml # 6 topic categories + genre taxonomy
├── prompts/ # LLM prompt templates (editable)
├── frontend/ # React web UI (placeholder)
├── tests/ # Test fixtures and test suites
│ └── fixtures/ # Sample data for testing
├── docker-compose.yml # Full stack definition
├── alembic.ini # Alembic configuration
├── .env.example # Environment variable template
└── chrysopedia-spec.md # Full project specification
```
---
## API Endpoints
| Method | Path | Description |
|--------|-----------------------------|---------------------------------|
| GET | `/health` | Health check with DB connectivity |
| GET | `/api/v1/health` | Lightweight health (no DB) |
| GET | `/api/v1/creators` | List all creators |
| GET | `/api/v1/creators/{slug}` | Get creator by slug |
| GET | `/api/v1/videos` | List all source videos |
---
## XPLTD Conventions
This project follows XPLTD infrastructure conventions:
- **Docker project name:** `xpltd_chrysopedia`
- **Bind mounts:** persistent data stored under `/vmPool/r/services/`
- **Network:** dedicated bridge `chrysopedia` (`172.24.0.0/24`)
- **PostgreSQL host port:** `5433` (avoids conflict with system PostgreSQL on `5432`)

148
tests/fixtures/sample_transcript.json vendored Normal file
View file

@ -0,0 +1,148 @@
{
"source_file": "Skope — Sound Design Masterclass pt1.mp4",
"creator_folder": "Skope",
"duration_seconds": 3847,
"segments": [
{
"start": 0.0,
"end": 4.52,
"text": "Hey everyone welcome back to part one of this sound design masterclass.",
"words": [
{ "word": "Hey", "start": 0.0, "end": 0.28 },
{ "word": "everyone", "start": 0.32, "end": 0.74 },
{ "word": "welcome", "start": 0.78, "end": 1.12 },
{ "word": "back", "start": 1.14, "end": 1.38 },
{ "word": "to", "start": 1.40, "end": 1.52 },
{ "word": "part", "start": 1.54, "end": 1.76 },
{ "word": "one", "start": 1.78, "end": 1.98 },
{ "word": "of", "start": 2.00, "end": 2.12 },
{ "word": "this", "start": 2.14, "end": 2.34 },
{ "word": "sound", "start": 2.38, "end": 2.68 },
{ "word": "design", "start": 2.72, "end": 3.08 },
{ "word": "masterclass", "start": 3.14, "end": 4.52 }
]
},
{
"start": 5.10,
"end": 12.84,
"text": "Today we're going to be looking at how to create really aggressive bass sounds using Serum.",
"words": [
{ "word": "Today", "start": 5.10, "end": 5.48 },
{ "word": "we're", "start": 5.52, "end": 5.74 },
{ "word": "going", "start": 5.78, "end": 5.98 },
{ "word": "to", "start": 6.00, "end": 6.12 },
{ "word": "be", "start": 6.14, "end": 6.28 },
{ "word": "looking", "start": 6.32, "end": 6.64 },
{ "word": "at", "start": 6.68, "end": 6.82 },
{ "word": "how", "start": 6.86, "end": 7.08 },
{ "word": "to", "start": 7.12, "end": 7.24 },
{ "word": "create", "start": 7.28, "end": 7.62 },
{ "word": "really", "start": 7.68, "end": 8.02 },
{ "word": "aggressive", "start": 8.08, "end": 8.72 },
{ "word": "bass", "start": 8.78, "end": 9.14 },
{ "word": "sounds", "start": 9.18, "end": 9.56 },
{ "word": "using", "start": 9.62, "end": 9.98 },
{ "word": "Serum", "start": 10.04, "end": 12.84 }
]
},
{
"start": 13.40,
"end": 22.18,
"text": "So the first thing I always do is start with the init preset and then I'll load up a basic wavetable.",
"words": [
{ "word": "So", "start": 13.40, "end": 13.58 },
{ "word": "the", "start": 13.62, "end": 13.78 },
{ "word": "first", "start": 13.82, "end": 14.12 },
{ "word": "thing", "start": 14.16, "end": 14.42 },
{ "word": "I", "start": 14.48, "end": 14.58 },
{ "word": "always", "start": 14.62, "end": 14.98 },
{ "word": "do", "start": 15.02, "end": 15.18 },
{ "word": "is", "start": 15.22, "end": 15.38 },
{ "word": "start", "start": 15.44, "end": 15.78 },
{ "word": "with", "start": 15.82, "end": 16.02 },
{ "word": "the", "start": 16.06, "end": 16.18 },
{ "word": "init", "start": 16.24, "end": 16.52 },
{ "word": "preset", "start": 16.58, "end": 17.02 },
{ "word": "and", "start": 17.32, "end": 17.48 },
{ "word": "then", "start": 17.52, "end": 17.74 },
{ "word": "I'll", "start": 17.78, "end": 17.98 },
{ "word": "load", "start": 18.04, "end": 18.32 },
{ "word": "up", "start": 18.36, "end": 18.52 },
{ "word": "a", "start": 18.56, "end": 18.64 },
{ "word": "basic", "start": 18.68, "end": 19.08 },
{ "word": "wavetable", "start": 19.14, "end": 22.18 }
]
},
{
"start": 23.00,
"end": 35.42,
"text": "What makes this technique work is the FM modulation from oscillator B. You want to set the ratio to something like 3.5 and then automate the depth.",
"words": [
{ "word": "What", "start": 23.00, "end": 23.22 },
{ "word": "makes", "start": 23.26, "end": 23.54 },
{ "word": "this", "start": 23.58, "end": 23.78 },
{ "word": "technique", "start": 23.82, "end": 24.34 },
{ "word": "work", "start": 24.38, "end": 24.68 },
{ "word": "is", "start": 24.72, "end": 24.88 },
{ "word": "the", "start": 24.92, "end": 25.04 },
{ "word": "FM", "start": 25.10, "end": 25.42 },
{ "word": "modulation", "start": 25.48, "end": 26.12 },
{ "word": "from", "start": 26.16, "end": 26.38 },
{ "word": "oscillator", "start": 26.44, "end": 27.08 },
{ "word": "B", "start": 27.14, "end": 27.42 },
{ "word": "You", "start": 28.02, "end": 28.22 },
{ "word": "want", "start": 28.26, "end": 28.52 },
{ "word": "to", "start": 28.56, "end": 28.68 },
{ "word": "set", "start": 28.72, "end": 28.98 },
{ "word": "the", "start": 29.02, "end": 29.14 },
{ "word": "ratio", "start": 29.18, "end": 29.58 },
{ "word": "to", "start": 29.62, "end": 29.76 },
{ "word": "something", "start": 29.80, "end": 30.22 },
{ "word": "like", "start": 30.26, "end": 30.48 },
{ "word": "3.5", "start": 30.54, "end": 31.02 },
{ "word": "and", "start": 31.32, "end": 31.48 },
{ "word": "then", "start": 31.52, "end": 31.74 },
{ "word": "automate", "start": 31.80, "end": 32.38 },
{ "word": "the", "start": 32.42, "end": 32.58 },
{ "word": "depth", "start": 32.64, "end": 35.42 }
]
},
{
"start": 36.00,
"end": 48.76,
"text": "Now I'm going to add some distortion. OTT is great for this. Crank it to like 60 percent and then back off the highs a bit with a shelf EQ.",
"words": [
{ "word": "Now", "start": 36.00, "end": 36.28 },
{ "word": "I'm", "start": 36.32, "end": 36.52 },
{ "word": "going", "start": 36.56, "end": 36.82 },
{ "word": "to", "start": 36.86, "end": 36.98 },
{ "word": "add", "start": 37.02, "end": 37.28 },
{ "word": "some", "start": 37.32, "end": 37.58 },
{ "word": "distortion", "start": 37.64, "end": 38.34 },
{ "word": "OTT", "start": 39.02, "end": 39.42 },
{ "word": "is", "start": 39.46, "end": 39.58 },
{ "word": "great", "start": 39.62, "end": 39.92 },
{ "word": "for", "start": 39.96, "end": 40.12 },
{ "word": "this", "start": 40.16, "end": 40.42 },
{ "word": "Crank", "start": 41.02, "end": 41.38 },
{ "word": "it", "start": 41.42, "end": 41.56 },
{ "word": "to", "start": 41.60, "end": 41.72 },
{ "word": "like", "start": 41.76, "end": 41.98 },
{ "word": "60", "start": 42.04, "end": 42.38 },
{ "word": "percent", "start": 42.42, "end": 42.86 },
{ "word": "and", "start": 43.12, "end": 43.28 },
{ "word": "then", "start": 43.32, "end": 43.54 },
{ "word": "back", "start": 43.58, "end": 43.84 },
{ "word": "off", "start": 43.88, "end": 44.08 },
{ "word": "the", "start": 44.12, "end": 44.24 },
{ "word": "highs", "start": 44.28, "end": 44.68 },
{ "word": "a", "start": 44.72, "end": 44.82 },
{ "word": "bit", "start": 44.86, "end": 45.08 },
{ "word": "with", "start": 45.14, "end": 45.38 },
{ "word": "a", "start": 45.42, "end": 45.52 },
{ "word": "shelf", "start": 45.58, "end": 45.96 },
{ "word": "EQ", "start": 46.02, "end": 48.76 }
]
}
]
}