From ba722304780d8864bbccd98058d3a8d6ef7efbdf Mon Sep 17 00:00:00 2001 From: jlightner Date: Sat, 4 Apr 2026 10:31:50 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20update=2010=20wiki=20pages=20with=20M02?= =?UTF-8?q?5=20feature=20content=20=E2=80=94=20notifications,=20rate=20lim?= =?UTF-8?q?iting,=20onboarding,=20transparency,=20export,=20fallback,=20mo?= =?UTF-8?q?bile,=20quality=20toolkit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- API-Surface.md | 29 +++++++++++++++++++++++++++++ Architecture.md | 4 +++- Chat-Engine.md | 30 ++++++++++++++++++++++++++++++ Configuration.md | 32 ++++++++++++++++++++++++++++++++ Data-Model.md | 40 ++++++++++++++++++++++++++++++++++++++++ Decisions.md | 9 +++++++++ Deployment.md | 24 ++++++++++++++++++++++++ Frontend.md | 39 +++++++++++++++++++++++++++++++++++++++ Home.md | 26 ++++++++++++++++++++++---- _Sidebar.md | 1 + 10 files changed, 229 insertions(+), 5 deletions(-) diff --git a/API-Surface.md b/API-Surface.md index 3d47540..1ff2749 100644 --- a/API-Surface.md +++ b/API-Surface.md @@ -200,6 +200,35 @@ All under prefix `/api/v1/admin/pipeline/`. **New endpoints should follow the `{items, total, offset, limit}` paginated pattern.** +## Notifications (M025/S01) + +| Method | Path | Auth | Notes | +|--------|------|------|-------| +| GET | `/api/v1/notifications/preferences` | Bearer JWT | Returns notification_preferences JSONB | +| PUT | `/api/v1/notifications/preferences` | Bearer JWT | Update digest frequency, enabled flags | +| GET | `/api/v1/notifications/unsubscribe?token=` | Public | JWT-signed token with expiry; renders styled HTML result | + +## Creator Transparency & Export (M025/S05, S07) + +| Method | Path | Auth | Notes | +|--------|------|------|-------| +| GET | `/api/v1/creator/transparency` | Bearer JWT | Full entity graph: technique pages, key moments, cross-refs, source videos, tags | +| GET | `/api/v1/creator/export` | Bearer JWT | GDPR-style ZIP download (12 data tables + export_metadata.json) | + +## Onboarding (M025/S03) + +| Method | Path | Auth | Notes | +|--------|------|------|-------| +| POST | `/api/v1/auth/onboarding-complete` | Bearer JWT | Sets `onboarding_completed = true` on User | + +## Usage & Rate Limiting (M025/S04) + +| Method | Path | Auth | Notes | +|--------|------|------|-------| +| GET | `/api/v1/admin/usage` | Admin | Token consumption: period totals, top creators/users, daily counts | + +Chat requests are rate-limited per-user (30/hr), per-IP (10/hr), and per-creator (60/hr) via Redis sliding-window. Returns HTTP 429 with `retry_after` header when exceeded. + ## Authentication JWT-based authentication added in M019. See [[Authentication]] for full details. diff --git a/Architecture.md b/Architecture.md index 161523c..4ece5f5 100644 --- a/Architecture.md +++ b/Architecture.md @@ -74,7 +74,7 @@ Chrysopedia is a self-hosted music production knowledge base that synthesizes te | Frontend | React 18 + TypeScript + Vite | | Backend | FastAPI + Celery + SQLAlchemy (async) | | Database | PostgreSQL 16 | -| Cache/Broker | Redis 7 (Celery broker + review mode toggle + classification cache) | +| Cache/Broker | Redis 7 (Celery broker + Beat scheduler + review mode toggle + classification cache + rate limit counters) | | Vector Store | Qdrant 1.13.2 | | Knowledge Graph | LightRAG (graph-based RAG, port 9621) | | Embeddings | Ollama (nomic-embed-text) via OpenAI-compatible /v1/embeddings | @@ -90,6 +90,7 @@ Chrysopedia is a self-hosted music production knowledge base that synthesizes te 4. **Storage:** Technique pages + key moments → PostgreSQL, embeddings → Qdrant 5. **Serving:** React SPA fetches from FastAPI, search queries hit Qdrant then PostgreSQL fallback 6. **Auth:** JWT-protected endpoints for creator consent management and admin features (see [[Authentication]]) +7. **Notifications:** Celery Beat runs daily email digest task (09:00 UTC) -- queries new content per followed creator, composes HTML, sends via SMTP. Deduplication via `EmailDigestLog` table (M025/S01) --- @@ -97,6 +98,7 @@ Chrysopedia is a self-hosted music production knowledge base that synthesizes te → PostgreSQL, embeddings → Qdrant 5. **Serving:** React SPA fetches from FastAPI, search queries hit Qdrant then PostgreSQL fallback 6. **Auth:** JWT-protected endpoints for creator consent management and admin features (see [[Authentication]]) +7. **Notifications:** Celery Beat runs daily email digest task (09:00 UTC) -- queries new content per followed creator, composes HTML, sends via SMTP. Deduplication via `EmailDigestLog` table (M025/S01) --- diff --git a/Chat-Engine.md b/Chat-Engine.md index 64450a9..9bc0392 100644 --- a/Chat-Engine.md +++ b/Chat-Engine.md @@ -225,6 +225,36 @@ End-to-end video metadata flow from search results through SSE to frontend sourc - `utils/formatTime.ts` — shared hour-aware time formatter used across timestamp badges and player controls. - Source cards now show: timestamp badge (links to `/watch/:id?t=N` when `start_time` defined) and video filename metadata. + +## LLM Fallback Resilience (M025/S08) + +ChatService maintains two AsyncOpenAI clients: primary (DGX endpoint) and fallback (Ollama). When the primary `create()` call fails with `APIConnectionError`, `APITimeoutError`, or `InternalServerError`, the entire streaming call is retried with the fallback client. The SSE `done` event includes `fallback_used: true/false` so the frontend and usage logging know which model actually served the response. + +- **Config:** `LLM_FALLBACK_URL` and `LLM_FALLBACK_MODEL` in docker-compose.yml +- **Logging:** `chat_llm_fallback` WARNING when primary fails and fallback activates +- **Usage tracking:** `ChatUsageLog.model` records actual model name (primary or fallback) + +## Refined System Prompt (M025/S09) + +The system prompt was rewritten from 5 lines to a structured template covering: + +- **Citation density:** Cite every factual claim inline with `[N]` markers +- **Response format:** Short paragraphs, bullet lists for step-by-step, bold key terms +- **Domain terminology:** Music production context awareness +- **Conflicting sources:** Present both perspectives with attribution +- **Response length:** 2-4 paragraphs default, adjust to query complexity + +Kept under 20 lines using markdown headers for structure. All 26 existing chat tests pass unchanged. + +## Chat Quality Evaluation Toolkit (M025/S09) + +A 5-dimension LLM-as-judge evaluation framework: + +- **Scorer** (`backend/pipeline/quality/chat_scorer.py`): Grades responses on citation_accuracy, response_structure, domain_expertise, source_grounding, personality_fidelity +- **Eval Harness** (`backend/pipeline/quality/chat_eval.py`): SSE-parsing runner that calls the live chat endpoint and feeds responses to the scorer +- **Test Suite** (`backend/pipeline/quality/fixtures/chat_test_suite.yaml`): 10 queries across technical, conceptual, creator-scoped, and cross-creator categories +- **CLI:** `python -m pipeline.quality chat_eval` subcommand for automated evaluation runs +- **Baseline report:** Documented in S09-QUALITY-REPORT.md with JSON results ## Key Files - `backend/chat_service.py` — ChatService with history load/save, retrieve-prompt-stream pipeline diff --git a/Configuration.md b/Configuration.md index 8175071..afbd632 100644 --- a/Configuration.md +++ b/Configuration.md @@ -145,6 +145,38 @@ LLM settings are configured per pipeline stage: Prompt templates are loaded from disk (`prompts/` directory) at runtime. SHA-256 hashes are tracked for reproducibility. + +## SMTP / Email Notifications (M025/S01) + +| Variable | Default | Notes | +|----------|---------|-------| +| `smtp_host` | `""` (empty) | SMTP server hostname. Empty = email disabled (graceful no-op) | +| `smtp_port` | `587` | SMTP port (587 for STARTTLS) | +| `smtp_user` | `""` | SMTP username | +| `smtp_password` | `""` | SMTP password | +| `smtp_from_email` | `""` | Sender email address | +| `smtp_from_name` | `"Chrysopedia"` | Sender display name | + +When `smtp_host` is empty, the digest task logs a warning and returns without sending. No emails are lost -- the next run picks up unsent content. + +## Rate Limiting (M025/S04) + +| Variable | Default | Notes | +|----------|---------|-------| +| `rate_limit_user_per_hour` | `30` | Max chat requests per authenticated user per hour | +| `rate_limit_ip_per_hour` | `10` | Max chat requests per IP per hour (anonymous) | +| `rate_limit_creator_per_hour` | `60` | Max chat requests per creator scope per hour | + +Rate limiter uses Redis sorted sets (sliding window). Fails open on Redis errors -- logs WARNING, allows request. + +## LLM Fallback (M025/S08) + +| Variable | Default | Notes | +|----------|---------|-------| +| `LLM_FALLBACK_URL` | `http://chrysopedia-ollama:11434/v1` | Fallback LLM endpoint (Ollama) | +| `LLM_FALLBACK_MODEL` | `qwen2.5:7b` | Fallback model name | + +ChatService auto-falls back from primary to fallback on `APIConnectionError`, `APITimeoutError`, or `InternalServerError`. Fallback activation is logged at WARNING and tracked in SSE `done` event and `ChatUsageLog`. ## Network - **Compose subnet:** 172.32.0.0/24 diff --git a/Data-Model.md b/Data-Model.md index 0194975..d12c0c9 100644 --- a/Data-Model.md +++ b/Data-Model.md @@ -252,6 +252,46 @@ Append-only versioned record of per-field consent changes. - **User passwords** are stored as bcrypt hashes via `bcrypt.hashpw()` - **Consent audit** uses version numbers assigned in application code (`max(version) + 1` per video_consent_id) + +## M025 Models + +### EmailDigestLog (M025/S01) + +| Field | Type | Notes | +|-------|------|-------| +| id | UUID PK | | +| user_id | FK -> User | | +| content_summary | JSONB | Digest content grouped by creator | +| sent_at | Timestamp | | + +Tracks every successful email send for deduplication. The digest task checks `MAX(sent_at)` per user to find new content since last send. + +### ChatUsageLog (M025/S04) + +| Field | Type | Notes | +|-------|------|-------| +| id | UUID PK | | +| user_id | FK -> User | Nullable (anonymous users) | +| client_ip | String | For IP-based rate limiting | +| creator_slug | String | Nullable | +| query | Text | User's chat query | +| prompt_tokens | Integer | | +| completion_tokens | Integer | | +| total_tokens | Integer | | +| cascade_tier | String | creator / domain / global / none | +| model | String | Actual model used (primary or fallback) | +| latency_ms | Integer | End-to-end response time | +| created_at | Timestamp | Indexed for time-range aggregation | + +Non-blocking INSERT -- errors logged but never block the SSE response. + +### User Model Changes (M025/S01, S03) + +| Field | Type | Notes | +|-------|------|-------| +| notification_preferences | JSONB | `{"digest_enabled": bool, "digest_frequency": "daily"|"weekly"}` | +| onboarding_completed | Boolean | Default false; set true via POST /auth/onboarding-complete | + --- *See also: [[Architecture]], [[API-Surface]], [[Pipeline]], [[Authentication]]* diff --git a/Decisions.md b/Decisions.md index 14e6835..c50766b 100644 --- a/Decisions.md +++ b/Decisions.md @@ -69,6 +69,15 @@ Architectural and pattern decisions made during Chrysopedia development. Append- | D043 | M023/S02 | Personality weight modulation strategy | 3-tier intensity (<0.4 subtle, 0.4–0.8 voice, ≥0.8 embody) with temperature scaling 0.3–0.5. **Superseded by D044.** | Initial stepped approach; replaced by continuous interpolation. | | D044 | M023/S04 | Personality weight modulation strategy (revision) | 5-tier continuous interpolation. Progressive field inclusion: <0.2 no personality; 0.2+ tone; 0.4+ descriptors; 0.6+ phrases (count scaled); 0.8+ vocabulary/style; 0.9+ summary. Temperature: 0.3 + weight × 0.2. | 3-tier step function had jarring transitions. Continuous interpolation with progressive field inclusion gives finer control. 0.0–0.19 dead zone ensures purely encyclopedic mode. | +## M025 Decisions + +| # | When | Decision | Choice | Rationale | +|---|------|----------|--------|-----------| +| D045 | M025/S01 | Signed unsubscribe token library | PyJWT (already a dependency) over itsdangerous | Avoids adding a new dependency; PyJWT supports exp claims natively for self-expiring tokens | +| D046 | M025/S10 | Sticky section bar for technique pages | Built ReadingHeader with sentinel-based IntersectionObserver | Sticky elements never leave viewport for standard IO detection; sentinel div pattern solves this reliably | +| D047 | M025/S04 | Rate limiter failure mode | Fail-open (allow request on Redis error) | Availability over strictness for a single-admin platform; rate limiting is cost protection, not security | +| D048 | M025/S08 | LLM fallback strategy | Catch-and-retry with secondary client on transient errors | Matches sync LLMClient pipeline pattern; propagates fallback_used through SSE and usage logging | + --- *See also: [[Architecture]], [[Development-Guide]]* diff --git a/Deployment.md b/Deployment.md index 5d6947d..4376af9 100644 --- a/Deployment.md +++ b/Deployment.md @@ -118,6 +118,30 @@ docker compose build chrysopedia-api && docker compose up -d chrysopedia-api chr docker compose restart chrysopedia-api chrysopedia-worker ``` + +## Email Digest Scheduling (M025/S01) + +The Celery worker runs with the `--beat` flag to enable periodic task scheduling: + +``` +command: celery -A worker worker --loglevel=info --beat +``` + +**Celery Beat schedule:** +- `send-digest-emails`: Runs daily at 09:00 UTC via `crontab(hour=9, minute=0)` +- Collocated with `worker.py` celery_app config (no separate beat container needed) + +## LLM Fallback Configuration (M025/S08) + +Add to the API service environment in `docker-compose.yml`: + +```yaml +environment: + LLM_FALLBACK_URL: http://chrysopedia-ollama:11434/v1 + LLM_FALLBACK_MODEL: qwen2.5:7b +``` + +The fallback endpoint is the local Ollama instance running on the same Docker network. No external dependency -- fallback works even when the primary DGX endpoint is unreachable. ## Port Mapping | Service | Container Port | Host Port | Binding | diff --git a/Frontend.md b/Frontend.md index 294899c..4ec15a3 100644 --- a/Frontend.md +++ b/Frontend.md @@ -193,6 +193,45 @@ API modules: Relative `/api/v1` base URL (nginx proxies to API container). + +## M025 Pages & Components + +### CreatorOnboarding (M025/S03) + +- **Route:** `/creator/onboarding` +- **File:** `frontend/src/pages/CreatorOnboarding.tsx` + `.module.css` +- **Auth:** ProtectedRoute (requires login) +- **Description:** 3-step wizard (Welcome -> Consent Setup -> Dashboard Tour). Step 2 fetches real consent data. Stepper UI with numbered circles and connecting lines. Responsive at 375px. +- **Trigger:** Login redirects here when `onboarding_completed === false` + +### AdminUsage (M025/S04) + +- **Route:** `/admin/usage` +- **File:** `frontend/src/pages/AdminUsage.tsx` + `.module.css` +- **Auth:** Admin only (via AdminDropdown menu) +- **Description:** Token usage dashboard with summary cards (today/week/month totals), CSS bar chart for daily activity, and breakdown tables (top creators, top users). +- **API:** `GET /api/v1/admin/usage` + +### CreatorTransparency (M025/S05) + +- **Route:** `/creator/transparency` +- **File:** `frontend/src/pages/CreatorTransparency.tsx` + `.module.css` +- **Auth:** ProtectedRoute (requires linked creator profile) +- **Description:** Full entity graph viewer with tag summary bar and 4 collapsible sections: Technique Pages, Key Moments (grouped by source video), Cross-References, Source Videos. Uses `grid-template-rows: 0fr/1fr` animation for collapse/expand. +- **Nav:** Sidebar link between Tiers and Posts on CreatorDashboard + +### ReadingHeader (M025/S10) + +- **File:** `frontend/src/components/ReadingHeader.tsx` +- **Description:** Sticky section bar on technique pages. Fixed-position bar slides in via CSS transform when user scrolls past the title. Uses sentinel-based IntersectionObserver with callback ref pattern. +- **Integration:** Rendered by TechniquePage.tsx, receives `activeId` from parent + +### Export My Data (M025/S07) + +- **Location:** Button on CreatorDashboard +- **API:** `exportCreatorData()` in `frontend/src/api/creator-dashboard.ts` +- **Pattern:** `fetch` with Bearer token -> `response.blob()` -> hidden anchor + `URL.createObjectURL` -> click -> revoke +- **UX:** Loading spinner during download, inline error display on failure ## CSS Architecture | Property | Value | diff --git a/Home.md b/Home.md index d45fa78..21e8d3f 100644 --- a/Home.md +++ b/Home.md @@ -8,7 +8,7 @@ Producers can search for specific techniques and find timestamped key moments, s - [[Architecture]] — System architecture, Docker services, network topology - [[Data-Model]] — SQLAlchemy models, relationships, enums -- [[API-Surface]] — All 70+ API endpoints grouped by domain +- [[API-Surface]] — All 80+ API endpoints grouped by domain - [[Frontend]] — Routes, components, hooks, CSS architecture - [[Pipeline]] — 6-stage LLM extraction pipeline, prompt system - [[Chat-Engine]] — Streaming Q&A with multi-turn memory @@ -17,7 +17,7 @@ Producers can search for specific techniques and find timestamped key moments, s - [[Search-Retrieval]] — LightRAG + Qdrant retrieval cascade - [[Deployment]] — Docker Compose setup, rebuild commands - [[Development-Guide]] — Local dev setup, common gotchas -- [[Decisions]] — Architectural decisions register (D001–D044) +- [[Decisions]] — Architectural decisions register (D001–D046) ## Features @@ -50,6 +50,24 @@ Producers can search for specific techniques and find timestamped key moments, s - **Inline Collapsible Player** — On technique pages between summary and body, with bibliography seek wiring and multi-source-video selector (M024/S02) - **Citation UX** — Timestamp badge links to `/watch/:id?t=N`, video filename on source cards, shared `chatCitations.tsx` + `formatTime.ts` utilities (M024/S05) +### Notifications & Cost Management (M025) +- **Email Digest Notifications** — Daily Celery Beat emails to followers when creators publish, with signed-token unsubscribe (M025/S01) +- **Rate Limiting** — Redis sliding-window per-user/IP/creator rate limiter for chat, fail-open design (M025/S04) +- **Usage Dashboard** — Admin token consumption tracking with period aggregation, top creators/users, daily charts (M025/S04) + +### Creator Experience (M025) +- **Onboarding Wizard** — 3-step welcome→consent→tour flow for new creators, triggered on first login (M025/S03) +- **AI Transparency Page** — Full entity graph viewer: technique pages, key moments, cross-references, source videos, tags (M025/S05) +- **Data Export** — GDPR-style ZIP download of all creator-owned content (12 tables + metadata) (M025/S07) + +### Resilience & Quality (M025) +- **LLM Fallback** — ChatService automatic primary→Ollama failover on connection/timeout/server errors (M025/S08) +- **Prompt Optimization** — Refined system prompt with citation density, structure, and domain guidance (M025/S09) +- **Chat Quality Toolkit** — 5-dimension LLM-as-judge scorer, SSE eval harness, 10-query test suite (M025/S09) +- **Mobile Responsiveness** — ≤400px safety-net breakpoints across all pages, zero horizontal overflow at 375px (M025/S02) +- **Reading Header** — Sticky section bar on technique pages with sentinel-based scroll detection (M025/S10) +- **Graph Evaluation** — NetworkX benchmark report with Neo4j migration plan at 50K/90K node thresholds (M025/S06) + ### Platform - **Authentication** — JWT with invite codes, admin/creator roles - **Consent System** — Per-video granular consent with audit trail @@ -57,7 +75,7 @@ Producers can search for specific techniques and find timestamped key moments, s ## Current Scale -- **83** technique pages +- **95** technique pages - **25–26** creators - **200** source videos - **7** topic categories @@ -78,4 +96,4 @@ Producers can search for specific techniques and find timestamped key moments, s --- -*Last updated: 2026-04-04 — M024 shorts publishing, embed support, timeline pins, auto-captioning, templates, citation UX +*Last updated: 2026-04-04 — M025 notifications, rate limiting, onboarding, transparency, export, fallback, mobile, quality toolkit diff --git a/_Sidebar.md b/_Sidebar.md index 3236c84..8589d28 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -1,6 +1,7 @@ ### Chrysopedia Wiki - [[Home]] +- [[Newcomer-Guide|Newcomer Guide]] **Architecture** - [[Architecture]]