From e77f8e0feacfe225c0378d4c3f5579a2edc935e9 Mon Sep 17 00:00:00 2001 From: jlightner Date: Sat, 4 Apr 2026 15:32:48 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20Updated=2010=20Forgejo=20wiki=20pages?= =?UTF-8?q?=20with=20M025=20feature=20documentation=20=E2=80=94=20n?= =?UTF-8?q?=E2=80=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - "Home.md" - "API-Surface.md" - "Data-Model.md" - "Frontend.md" - "Configuration.md" - "Chat-Engine.md" - "Deployment.md" - "Decisions.md" GSD-Task: S11/T01 --- .gsd/milestones/M025/M025-ROADMAP.md | 2 +- .../milestones/M025/slices/S10/S10-SUMMARY.md | 97 +++++++++++++ .gsd/milestones/M025/slices/S10/S10-UAT.md | 64 +++++++++ .../M025/slices/S10/tasks/T02-VERIFY.json | 18 +++ .gsd/milestones/M025/slices/S11/S11-PLAN.md | 135 +++++++++++++++++- .../M025/slices/S11/S11-RESEARCH.md | 89 ++++++++++++ .../M025/slices/S11/tasks/T01-PLAN.md | 93 ++++++++++++ .../M025/slices/S11/tasks/T01-SUMMARY.md | 93 ++++++++++++ .../M025/slices/S11/tasks/T02-PLAN.md | 95 ++++++++++++ 9 files changed, 684 insertions(+), 2 deletions(-) create mode 100644 .gsd/milestones/M025/slices/S10/S10-SUMMARY.md create mode 100644 .gsd/milestones/M025/slices/S10/S10-UAT.md create mode 100644 .gsd/milestones/M025/slices/S10/tasks/T02-VERIFY.json create mode 100644 .gsd/milestones/M025/slices/S11/S11-RESEARCH.md create mode 100644 .gsd/milestones/M025/slices/S11/tasks/T01-PLAN.md create mode 100644 .gsd/milestones/M025/slices/S11/tasks/T01-SUMMARY.md create mode 100644 .gsd/milestones/M025/slices/S11/tasks/T02-PLAN.md diff --git a/.gsd/milestones/M025/M025-ROADMAP.md b/.gsd/milestones/M025/M025-ROADMAP.md index ddee62b..27cca31 100644 --- a/.gsd/milestones/M025/M025-ROADMAP.md +++ b/.gsd/milestones/M025/M025-ROADMAP.md @@ -15,5 +15,5 @@ Production hardening, mobile polish, creator onboarding, and formal validation. | S07 | [A] Data Export (GDPR-Style) | medium | — | ✅ | Creator downloads a ZIP with all derived content, entities, and relationships | | S08 | [B] Load Testing + Fallback Resilience | medium | — | ✅ | 10 concurrent chat sessions maintain acceptable latency. DGX down → Ollama fallback works. | | S09 | [B] Prompt Optimization Pass | low | — | ✅ | Chat quality reviewed across creators. Personality fidelity assessed. | -| S10 | Requirement Validation (R015, R037-R041) | low | — | ⬜ | R015, R037, R038, R039, R041 formally validated and signed off | +| S10 | Requirement Validation (R015, R037-R041) | low | — | ✅ | R015, R037, R038, R039, R041 formally validated and signed off | | S11 | Forgejo KB Final — Complete Documentation | low | S01, S02, S03, S04, S05, S06, S07, S08, S09, S10 | ⬜ | Forgejo wiki complete with newcomer onboarding guide covering entire platform | diff --git a/.gsd/milestones/M025/slices/S10/S10-SUMMARY.md b/.gsd/milestones/M025/slices/S10/S10-SUMMARY.md new file mode 100644 index 0000000..7f6327c --- /dev/null +++ b/.gsd/milestones/M025/slices/S10/S10-SUMMARY.md @@ -0,0 +1,97 @@ +--- +id: S10 +parent: M025 +milestone: M025 +provides: + - R015 validated + - R037 validated + - R038 validated + - R039 validated + - R041 validated + - ReadingHeader component +requires: + [] +affects: + - S11 +key_files: + - frontend/src/components/ReadingHeader.tsx + - frontend/src/pages/TechniquePage.tsx + - frontend/src/App.css + - .gsd/REQUIREMENTS.md +key_decisions: + - Sentinel-based scroll detection for sticky reading header (sticky elements never leave viewport for IntersectionObserver) + - Callback ref pattern for IntersectionObserver targeting conditionally-rendered elements +patterns_established: + - Sentinel div + IntersectionObserver for detecting scroll-past-element events on sticky/fixed elements +observability_surfaces: + - none +drill_down_paths: + - .gsd/milestones/M025/slices/S10/tasks/T01-SUMMARY.md + - .gsd/milestones/M025/slices/S10/tasks/T02-SUMMARY.md +duration: "" +verification_result: passed +completed_at: 2026-04-04T15:23:14.940Z +blocker_discovered: false +--- + +# S10: Requirement Validation (R015, R037-R041) + +**Formally validated five requirements (R015, R037, R038, R039, R041) via browser testing against live ub01:8096 deployment, built ReadingHeader component for R041, and updated all from active to validated.** + +## What Happened + +This slice had two tasks: build the ReadingHeader sticky bar component (T01, satisfying R041) and then browser-validate all five target requirements against the live deployment (T02). + +T01 created `ReadingHeader.tsx` — a fixed-position bar that slides in via CSS transform when the user scrolls past the technique page title. Uses a sentinel-based IntersectionObserver with callback ref pattern (direct observation of sticky elements doesn't work since they never leave the viewport). Deployed to ub01 and verified at desktop and 375px mobile viewports. + +T02 ran browser-based validation for each requirement: +- **R015 (30-Second Retrieval):** Homepage → search "snare" → technique page content visible in ~9 seconds, well under the 30s target. +- **R037 (Landing Page Visual Consistency):** Verified at 1280px (3-column card alignment, stats scorecard with count-up animation) and 375px (single-column, no overflow). +- **R038 (Pipeline Admin UI Fixes):** Collapse toggle works on Recent Activity, text filter shows multiple creators (KOAN Sound, COPYCATT, etc.), mobile layout has no vertical text. +- **R039 (Brand Minimum):** Favicon, OG tags, and logo all present in DOM. HTTP 200 for /favicon.svg and /og-image.png. +- **R041 (Sticky Reading Header):** `.reading-header--visible` class applied on scroll past title, responsive at 375px. Only renders on v2 body_sections_format pages (84 of 95). + +All five requirements updated from active → validated in REQUIREMENTS.md, bringing the total validated count from 32 to 37. + +## Verification + +All five requirements confirmed validated via grep: `for r in R015 R037 R038 R039 R041; do grep -A2 "^## $r " .gsd/REQUIREMENTS.md | grep 'Status:'; done` returns `**Status:** validated` for each. Browser evidence captured for each requirement across desktop and mobile viewports. TypeScript compiles clean, Docker build succeeds. + +## Requirements Advanced + +None. + +## Requirements Validated + +- R015 — Browser timed test: homepage → search 'snare' → technique page visible in ~9 seconds (target: 30s) +- R037 — Browser screenshots at 1280px and 375px: card alignment, stats scorecard count-up, no overflow +- R038 — Browser: collapse toggle works, text filter shows 6+ creators, mobile 375px no vertical text +- R039 — DOM: link[rel=icon], meta[og:title], meta[og:image], logo SVG. HTTP 200 for /favicon.svg and /og-image.png +- R041 — Browser: .reading-header--visible on scroll past title, responsive at 375px. CSS transition slide-in confirmed. + +## New Requirements Surfaced + +None. + +## Requirements Invalidated or Re-scoped + +None. + +## Deviations + +R038 validation used text filter (the actual UI) rather than a separate creator dropdown mentioned in the plan. R041 tested only on v2 pages (84/95 total). Both are accurate reflections of the actual implementation. + +## Known Limitations + +11 legacy v1 technique pages lack ReadingHeader (they use old body_sections format). Pre-existing activeId scroll-spy IntersectionObserver bug affects section name display in both ToC and ReadingHeader. + +## Follow-ups + +Fix activeId scroll-spy IntersectionObserver so section name updates correctly in ReadingHeader and ToC. Consider migrating remaining 11 v1 pages to v2 format. + +## Files Created/Modified + +- `frontend/src/components/ReadingHeader.tsx` — New component: sticky reading header with CSS transform slide-in/out +- `frontend/src/pages/TechniquePage.tsx` — Added sentinel-based IntersectionObserver and ReadingHeader integration +- `frontend/src/App.css` — Added Reading Header CSS section (~40px bar, transitions, mobile styles) +- `.gsd/REQUIREMENTS.md` — Updated R015, R037, R038, R039, R041 from active to validated with evidence diff --git a/.gsd/milestones/M025/slices/S10/S10-UAT.md b/.gsd/milestones/M025/slices/S10/S10-UAT.md new file mode 100644 index 0000000..99390a2 --- /dev/null +++ b/.gsd/milestones/M025/slices/S10/S10-UAT.md @@ -0,0 +1,64 @@ +# S10: Requirement Validation (R015, R037-R041) — UAT + +**Milestone:** M025 +**Written:** 2026-04-04T15:23:14.941Z + +## UAT: Requirement Validation (R015, R037-R041) + +### Preconditions +- Chrysopedia deployed and running at ub01:8096 +- At least one v2 technique page exists (e.g., /techniques/drop-first-production-copycatt) +- Pipeline admin accessible at /admin/pipeline + +### Test 1: R015 — 30-Second Retrieval Target +1. Open http://ub01:8096 in a browser +2. Start a timer +3. Type "snare" in the search bar +4. Click the first search result +5. **Expected:** Technique page content (h1 heading, body sections) visible +6. **Expected:** Total time from step 2 to content visible < 30 seconds + +### Test 2: R037 — Landing Page Visual Consistency +1. Open http://ub01:8096 at 1280px viewport width +2. **Expected:** 3-column card grid aligned, stats scorecard visible with count-up animation (numbers animate from 0) +3. **Expected:** Consistent spacing between sections, no duplicate CSS rule artifacts +4. Resize to 375px viewport +5. **Expected:** Single-column layout, no horizontal overflow, stats stack vertically + +### Test 3: R038 — Pipeline Admin UI Fixes +1. Navigate to http://ub01:8096/admin/pipeline +2. Locate "Recent Activity" section with collapse toggle +3. Click the toggle +4. **Expected:** Section collapses/expands without flicker +5. Check the creator/text filter +6. **Expected:** Multiple creators available (KOAN Sound, COPYCATT, Current Value, etc.) +7. Resize to 375px +8. **Expected:** Job cards display in single column, no vertical/rotated text + +### Test 4: R039 — Brand Minimum +1. Open http://ub01:8096 +2. Inspect DOM for `link[rel='icon']` +3. **Expected:** href contains "favicon" +4. Inspect DOM for `meta[property='og:title']` and `meta[property='og:image']` +5. **Expected:** Both present with non-empty content attributes +6. Check header area +7. **Expected:** SVG logo visible next to "Chrysopedia" text +8. Run: `curl -sf -o /dev/null -w "%{http_code}" http://ub01:8096/favicon.svg` +9. **Expected:** Returns 200 +10. Run: `curl -sf -o /dev/null -w "%{http_code}" http://ub01:8096/og-image.png` +11. **Expected:** Returns 200 + +### Test 5: R041 — Sticky Reading Header +1. Navigate to a v2 technique page with 4+ sections (e.g., /techniques/drop-first-production-copycatt) +2. Scroll down past the title bar +3. **Expected:** A thin sticky header slides in from top with truncated article title +4. **Expected:** Element has class `.reading-header--visible` +5. Scroll back to top +6. **Expected:** Reading header slides out (hidden) +7. Resize to 375px +8. Scroll past title +9. **Expected:** Reading header renders at full width, text truncated appropriately + +### Edge Cases +- **v1 pages:** Navigate to a v1-format technique page. ReadingHeader should NOT appear (only renders for v2 body_sections_format). +- **Short pages:** On a technique page with only 1-2 sections, reading header still appears on scroll-past-title but section name may show only the single section. diff --git a/.gsd/milestones/M025/slices/S10/tasks/T02-VERIFY.json b/.gsd/milestones/M025/slices/S10/tasks/T02-VERIFY.json new file mode 100644 index 0000000..8c46304 --- /dev/null +++ b/.gsd/milestones/M025/slices/S10/tasks/T02-VERIFY.json @@ -0,0 +1,18 @@ +{ + "schemaVersion": 1, + "taskId": "T02", + "unitId": "M025/S10/T02", + "timestamp": 1775316124591, + "passed": false, + "discoverySource": "task-plan", + "checks": [ + { + "command": "grep -c 'validated' .gsd/REQUIREMENTS.md shows 5 more validated requirements than before (R015, R037, R038, R039, R041 all show Status: validated)", + "exitCode": 2, + "durationMs": 9, + "verdict": "fail" + } + ], + "retryAttempt": 1, + "maxRetries": 2 +} diff --git a/.gsd/milestones/M025/slices/S11/S11-PLAN.md b/.gsd/milestones/M025/slices/S11/S11-PLAN.md index 4ee32c4..dea7abf 100644 --- a/.gsd/milestones/M025/slices/S11/S11-PLAN.md +++ b/.gsd/milestones/M025/slices/S11/S11-PLAN.md @@ -1,6 +1,139 @@ # S11: Forgejo KB Final — Complete Documentation -**Goal:** Complete Forgejo KB with full documentation and newcomer onboarding guide +**Goal:** Forgejo wiki complete with all M025 features documented and a newcomer onboarding guide covering the entire platform. **Demo:** After this: Forgejo wiki complete with newcomer onboarding guide covering entire platform ## Tasks +- [x] **T01: Updated 10 Forgejo wiki pages with M025 feature documentation — notifications, rate limiting, onboarding, transparency, export, fallback, prompt optimization, mobile, quality toolkit** — Update ~10 existing wiki pages in the Forgejo wiki with M025 feature documentation. All work happens via SSH to ub01, editing files in a fresh wiki clone, then committing and pushing. + +Content sources are the S01-S10 slice summaries from M025. The research doc identifies exactly which pages need updates and what content goes where. + +**CRITICAL:** Never use the Forgejo wiki PATCH API — it corrupts pages. Only use git clone/edit/push. + +**Wiki clone setup:** `cd /tmp && git clone https://git.xpltd.co/xpltdco/chrysopedia.wiki.git chrysopedia-wiki-m025` on ub01, or `cd /tmp/chrysopedia-wiki-m024 && git pull` to reuse the existing clone. + +**Pages to update (10):** + +1. **Home.md** (~81 lines) — Add M025 feature summary section listing: email digest notifications, rate limiting + cost management, creator onboarding wizard, AI transparency page, GDPR-style data export, LLM fallback resilience, prompt optimization, mobile responsiveness pass, requirement validation. Update any counts (page count, feature count). + +2. **API-Surface.md** (~216 lines) — Add new endpoints: + - `GET/PUT /notifications/preferences` (auth required) + - `GET /notifications/unsubscribe?token=` (public, JWT signed) + - `POST /auth/onboarding-complete` (auth required) + - `GET /creator/transparency` (auth required) + - `GET /creator/export` (auth required, returns ZIP) + - `GET /admin/usage` (admin required) + +3. **Data-Model.md** (~298 lines) — Add models: + - `EmailDigestLog` (id, user_id, content_summary JSONB, sent_at) + - `ChatUsageLog` (id, user_id, client_ip, creator_slug, query, prompt_tokens, completion_tokens, total_tokens, cascade_tier, model, latency_ms, created_at) + - `notification_preferences` JSONB column on User + - `onboarding_completed` boolean column on User + +4. **Frontend.md** (~235 lines) — Add pages/components: + - CreatorOnboarding (3-step wizard at /creator/onboarding) + - AdminUsage (token usage dashboard at /admin/usage) + - CreatorTransparency (entity viewer at /creator/transparency) + - ReadingHeader (sticky section bar on technique pages) + - Export My Data button on CreatorDashboard + +5. **Configuration.md** (~155 lines) — Add environment variables: + - SMTP: smtp_host, smtp_port, smtp_user, smtp_password, smtp_from_email, smtp_from_name + - Rate limits: rate_limit_user_per_hour, rate_limit_ip_per_hour, rate_limit_creator_per_hour + - LLM fallback: LLM_FALLBACK_URL, LLM_FALLBACK_MODEL + +6. **Chat-Engine.md** (~254 lines) — Add: + - Fallback resilience section: primary→Ollama automatic failover on connection/timeout/server errors + - Refined system prompt details: citation density, response structure, domain terminology + - Quality evaluation toolkit mention (chat_scorer, chat_eval, test suite) + +7. **Deployment.md** (~143 lines) — Add: + - Celery Beat: --beat flag on worker command for email digest scheduling + - LLM fallback env vars in docker-compose.yml + +8. **Decisions.md** (~74 lines) — Append: + - D045: PyJWT for signed unsubscribe tokens + - D046: ReadingHeader implementation over accepting partial sticky bar + +9. **_Sidebar.md** (~30 lines) — Add Newcomer Guide link + +10. **Architecture.md** (~104 lines) — Mention email digest as async subsystem (Celery Beat scheduled task) + - Estimate: 45m + - Files: Home.md (on ub01 wiki clone), API-Surface.md, Data-Model.md, Frontend.md, Configuration.md, Chat-Engine.md, Deployment.md, Decisions.md, _Sidebar.md, Architecture.md + - Verify: ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && git log --oneline -1' shows M025 commit AND ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && git diff HEAD~1 --stat' shows ~10 files changed +- [ ] **T02: Create Newcomer Guide and push all wiki changes** — Create the Newcomer-Guide.md page — the primary deliverable of this slice — and push all wiki changes (both T01 updates and this new page) to the Forgejo wiki. + +**CRITICAL:** Never use the Forgejo wiki PATCH API — it corrupts pages. Only use git push. + +**Wiki clone location:** `/tmp/chrysopedia-wiki-m025` on ub01 (set up by T01). + +**Newcomer-Guide.md structure (≥8 sections):** + +1. **What is Chrysopedia** — 3-sentence pitch: AI-powered knowledge base for music production techniques, built from video content by creators. Extracts, structures, and makes searchable the knowledge embedded in production tutorials and livestreams. + +2. **How Content Flows** — Simplified pipeline overview: + - Video transcripts land in a watched folder + - 6-stage pipeline: transcript segmentation → key moment extraction → classification/tagging → technique page synthesis → embedding/indexing → (optional) highlight detection + - Results: structured technique pages, searchable key moments, cross-references between creators + +3. **Using the Web UI** — End-user features: + - Search: typeahead search from homepage or Cmd+K anywhere, semantic + keyword + - Browse Topics: 7 top-level categories → sub-topics → technique pages grouped by creator + - Browse Creators: filterable list with genre tags, randomized default sort + - Technique Pages: study guide prose, key moments index, related techniques, table of contents + - Chat: creator-scoped AI chat with citation, cascade retrieval (creator→domain→global) + - Reading Header: sticky section bar on technique pages when scrolling + +4. **For Creators** — Creator-specific features: + - Registration and onboarding wizard (3-step: welcome, consent, tour) + - Consent dashboard: control which content types are published + - Creator dashboard: overview stats, recent posts, technique pages + - Transparency page: view all entities, relationships, technique pages derived from content + - Data export: GDPR-style ZIP download of all derived content + - Notification preferences: email digest frequency control + - Personality profiles: 5-tier system for chat persona customization + +5. **For Admins** — Administrative features: + - Review queue: approve/reject/edit key moments by source video + - Pipeline admin: monitor processing stages, filter by creator/status + - Usage dashboard: token consumption tracking, top creators/users, daily stats + - Audit log: all administrative actions tracked + - User management: role assignment, account status + +6. **Adding New Content** — Content ingestion workflow: + - Prepare video transcripts (Whisper large-v3 on RTX 4090) + - Place transcript JSON in watched folder + - Pipeline processes automatically through all stages + - Review queue for quality control before publishing + +7. **Infrastructure & Deployment** — High-level deployment info: + - Docker Compose stack on ub01 + - Services: API, Worker (Celery + Beat), Web (nginx), PostgreSQL, Redis, Qdrant, Ollama, LightRAG + - LLM: primary DGX endpoint with automatic Ollama fallback + - Link to Deployment and Development-Guide wiki pages + +8. **Where to Learn More** — Cross-reference index linking to every wiki page by topic: + - Architecture overview → Architecture + - API reference → API-Surface + - Data model → Data-Model + - Search & retrieval → Search-Retrieval + - Chat engine → Chat-Engine + - Pipeline stages → Pipeline + - Authentication → Authentication + - Configuration → Configuration + - Monitoring → Monitoring + - etc. + +**After creating the file, push all changes:** +```bash +ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && git add . && git commit -m "docs: M025 wiki update — notifications, rate limiting, onboarding, transparency, export, fallback, mobile, newcomer guide" && git push' +``` + +**Verify page count via API:** +```bash +ssh ub01 'curl -s "https://git.xpltd.co/api/v1/repos/xpltdco/chrysopedia/wiki/pages?limit=50" | python3 -c "import json,sys; pages=json.load(sys.stdin); print(len(pages), chr(39) + chr(39).join([]) + \"pages\")"' +``` +Expect ≥21 pages. + - Estimate: 30m + - Files: Newcomer-Guide.md (on ub01 wiki clone) + - Verify: ssh ub01 'curl -s "https://git.xpltd.co/api/v1/repos/xpltdco/chrysopedia/wiki/pages?limit=50" | python3 -c "import json,sys; pages=json.load(sys.stdin); print(len(pages))"' returns ≥21 AND ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && test -f Newcomer-Guide.md && grep -c "^## " Newcomer-Guide.md' returns ≥8 diff --git a/.gsd/milestones/M025/slices/S11/S11-RESEARCH.md b/.gsd/milestones/M025/slices/S11/S11-RESEARCH.md new file mode 100644 index 0000000..2bc8166 --- /dev/null +++ b/.gsd/milestones/M025/slices/S11/S11-RESEARCH.md @@ -0,0 +1,89 @@ +# S11 Research: Forgejo KB Final — Complete Documentation + +## Summary + +Straightforward wiki update slice. The pattern is established across M018/S02, M020/S07, M023/S05, and M024 — git clone wiki on ub01, edit markdown files, commit and push. No new technology, no ambiguous requirements. + +**Current state:** 20 wiki pages (19 content + _Sidebar) at `https://git.xpltd.co/xpltdco/chrysopedia/wiki/`. Last updated in M024 (commit 0451abf). The wiki clone at `/tmp/chrysopedia-wiki-m024` on ub01 is current and has git credential store configured for push. + +**Deliverable:** Update existing pages with M025 features + create a newcomer onboarding guide covering the entire platform. + +## Recommendation + +Two tasks: +1. **Update existing wiki pages** with M025 feature content (notifications, rate limiting, onboarding, transparency, export, fallback, prompt optimization, mobile responsiveness) +2. **Create Newcomer-Guide.md** — the primary deliverable — a walkthrough from zero to productive covering the entire platform + +## Implementation Landscape + +### Existing Wiki Pages (20 total) + +| Page | Lines | M025 Updates Needed | +|------|-------|-------------------| +| Home.md | 81 | Add M025 feature summary, update counts | +| API-Surface.md | 216 | Add ~10 new endpoints: notifications (3), transparency (1), export (1), admin/usage (1), onboarding-complete (1) | +| Data-Model.md | 298 | Add EmailDigestLog, ChatUsageLog models, onboarding_completed column | +| Frontend.md | 235 | Add CreatorOnboarding, AdminUsage, CreatorTransparency pages, ReadingHeader component | +| Configuration.md | 155 | Add SMTP settings (6 vars), rate limit settings (3 vars), LLM fallback settings (2 vars) | +| Chat-Engine.md | 254 | Add fallback resilience section, prompt optimization details | +| Deployment.md | 143 | Add Celery Beat --beat flag, email digest scheduling | +| Decisions.md | 74 | Add D045 (PyJWT for unsubscribe), D046 (ReadingHeader implementation) | +| _Sidebar.md | 30 | Add Newcomer Guide link, possibly Notifications link | +| Architecture.md | 104 | Minor: mention email digest as new async subsystem | + +Pages that DON'T need M025 updates: Player, Impersonation, Authentication, Search-Retrieval, Pipeline, Highlights, Personality-Profiles, Monitoring, Agent-Context, Development-Guide. + +### New Pages + +| Page | Purpose | +|------|---------| +| **Newcomer-Guide.md** | Zero-to-productive walkthrough: what the platform is, how content flows, how to use the UI, how to add content, how to administer. Cross-links to all other wiki pages. This is the primary slice deliverable. | + +### Wiki Update Method + +- **Clone:** Fresh `git clone https://git.xpltd.co/xpltdco/chrysopedia.wiki.git /tmp/chrysopedia-wiki-m025` on ub01 (or reuse existing clone with `git pull`) +- **Auth:** `git config credential.helper store` already configured with token in `~/.git-credentials` +- **Edit:** Modify .md files directly on ub01 +- **Push:** `git add . && git commit -m "docs: M025 ..." && git push` +- **Verify:** `curl -s .../wiki/pages?limit=50 | python3 -c "..."` to confirm page count +- **NEVER use Forgejo PATCH API** per KNOWLEDGE.md (corrupts pages to unnamed.md) + +### Content Sources for Updates + +All content comes from the preloaded S01-S10 summaries above. Key details: + +- **S01:** SMTP config (6 vars), EmailDigestLog model, notification_preferences JSONB on User, Celery Beat schedule, PyJWT signed unsubscribe, 3 API endpoints +- **S02:** ≤400px safety-net breakpoint pattern, hidden-scrollbar horizontal scroll pattern +- **S03:** onboarding_completed flag on User, 3-step wizard, POST /auth/onboarding-complete +- **S04:** Redis sliding-window rate limiter, ChatUsageLog model, GET /admin/usage, 3 config vars +- **S05:** GET /creator/transparency, selectinload chains, CSS grid-template-rows animation +- **S06:** Graph backend evaluation report (NetworkX recommendation, 50K/90K thresholds) +- **S07:** GET /creator/export (ZIP), GDPR-style data download +- **S08:** LLM fallback (primary→Ollama), LLM_FALLBACK_URL/MODEL config, load test script +- **S09:** Chat quality evaluation toolkit, refined system prompt, quality baseline report +- **S10:** ReadingHeader component, sentinel-based IntersectionObserver, R015/R037-R041 validated + +### Newcomer Guide Structure (recommended) + +1. **What is Chrysopedia** — 3-sentence pitch +2. **How Content Flows** — simplified version of the 6-stage pipeline diagram from README +3. **Using the Web UI** — search, browse topics/creators, technique pages, chat +4. **For Creators** — registration, onboarding, consent, dashboard, transparency, export +5. **For Admins** — review queue, pipeline admin, usage dashboard, audit log +6. **Adding New Content** — transcript preparation, folder watcher, pipeline processing +7. **Development Setup** — link to Development-Guide and Deployment pages +8. **Where to Learn More** — links to every wiki page by topic + +### Constraints + +- All work must happen via SSH to ub01 (`ssh ub01 'cd /tmp/... && ...'`) +- Git credential store handles auth — no manual token management needed +- The `Agent-Context.-.md` filename has a weird period-dash pattern (likely from Forgejo API normalization of spaces) — don't rename it +- Existing page structure/formatting should be preserved; append M025 sections to existing pages + +### Verification + +- `curl -s "https://git.xpltd.co/api/v1/repos/xpltdco/chrysopedia/wiki/pages?limit=50" | python3 -c "import json,sys; pages=json.load(sys.stdin); print(len(pages),'pages')"` — expect ≥21 (20 existing + Newcomer-Guide) +- `git log --oneline -1` confirms push succeeded +- `git diff HEAD~1 --stat` confirms expected files changed +- Spot-check: curl a couple updated pages and verify M025 content present diff --git a/.gsd/milestones/M025/slices/S11/tasks/T01-PLAN.md b/.gsd/milestones/M025/slices/S11/tasks/T01-PLAN.md new file mode 100644 index 0000000..fea162e --- /dev/null +++ b/.gsd/milestones/M025/slices/S11/tasks/T01-PLAN.md @@ -0,0 +1,93 @@ +--- +estimated_steps: 40 +estimated_files: 10 +skills_used: [] +--- + +# T01: Update existing wiki pages with M025 feature content + +Update ~10 existing wiki pages in the Forgejo wiki with M025 feature documentation. All work happens via SSH to ub01, editing files in a fresh wiki clone, then committing and pushing. + +Content sources are the S01-S10 slice summaries from M025. The research doc identifies exactly which pages need updates and what content goes where. + +**CRITICAL:** Never use the Forgejo wiki PATCH API — it corrupts pages. Only use git clone/edit/push. + +**Wiki clone setup:** `cd /tmp && git clone https://git.xpltd.co/xpltdco/chrysopedia.wiki.git chrysopedia-wiki-m025` on ub01, or `cd /tmp/chrysopedia-wiki-m024 && git pull` to reuse the existing clone. + +**Pages to update (10):** + +1. **Home.md** (~81 lines) — Add M025 feature summary section listing: email digest notifications, rate limiting + cost management, creator onboarding wizard, AI transparency page, GDPR-style data export, LLM fallback resilience, prompt optimization, mobile responsiveness pass, requirement validation. Update any counts (page count, feature count). + +2. **API-Surface.md** (~216 lines) — Add new endpoints: + - `GET/PUT /notifications/preferences` (auth required) + - `GET /notifications/unsubscribe?token=` (public, JWT signed) + - `POST /auth/onboarding-complete` (auth required) + - `GET /creator/transparency` (auth required) + - `GET /creator/export` (auth required, returns ZIP) + - `GET /admin/usage` (admin required) + +3. **Data-Model.md** (~298 lines) — Add models: + - `EmailDigestLog` (id, user_id, content_summary JSONB, sent_at) + - `ChatUsageLog` (id, user_id, client_ip, creator_slug, query, prompt_tokens, completion_tokens, total_tokens, cascade_tier, model, latency_ms, created_at) + - `notification_preferences` JSONB column on User + - `onboarding_completed` boolean column on User + +4. **Frontend.md** (~235 lines) — Add pages/components: + - CreatorOnboarding (3-step wizard at /creator/onboarding) + - AdminUsage (token usage dashboard at /admin/usage) + - CreatorTransparency (entity viewer at /creator/transparency) + - ReadingHeader (sticky section bar on technique pages) + - Export My Data button on CreatorDashboard + +5. **Configuration.md** (~155 lines) — Add environment variables: + - SMTP: smtp_host, smtp_port, smtp_user, smtp_password, smtp_from_email, smtp_from_name + - Rate limits: rate_limit_user_per_hour, rate_limit_ip_per_hour, rate_limit_creator_per_hour + - LLM fallback: LLM_FALLBACK_URL, LLM_FALLBACK_MODEL + +6. **Chat-Engine.md** (~254 lines) — Add: + - Fallback resilience section: primary→Ollama automatic failover on connection/timeout/server errors + - Refined system prompt details: citation density, response structure, domain terminology + - Quality evaluation toolkit mention (chat_scorer, chat_eval, test suite) + +7. **Deployment.md** (~143 lines) — Add: + - Celery Beat: --beat flag on worker command for email digest scheduling + - LLM fallback env vars in docker-compose.yml + +8. **Decisions.md** (~74 lines) — Append: + - D045: PyJWT for signed unsubscribe tokens + - D046: ReadingHeader implementation over accepting partial sticky bar + +9. **_Sidebar.md** (~30 lines) — Add Newcomer Guide link + +10. **Architecture.md** (~104 lines) — Mention email digest as async subsystem (Celery Beat scheduled task) + +## Inputs + +- ``.gsd/milestones/M025/slices/S01/S01-SUMMARY.md` — email digest feature details` +- ``.gsd/milestones/M025/slices/S02/S02-SUMMARY.md` — mobile responsiveness details` +- ``.gsd/milestones/M025/slices/S03/S03-SUMMARY.md` — onboarding flow details` +- ``.gsd/milestones/M025/slices/S04/S04-SUMMARY.md` — rate limiting details` +- ``.gsd/milestones/M025/slices/S05/S05-SUMMARY.md` — transparency page details` +- ``.gsd/milestones/M025/slices/S06/S06-SUMMARY.md` — graph evaluation details` +- ``.gsd/milestones/M025/slices/S07/S07-SUMMARY.md` — data export details` +- ``.gsd/milestones/M025/slices/S08/S08-SUMMARY.md` — fallback resilience details` +- ``.gsd/milestones/M025/slices/S09/S09-SUMMARY.md` — prompt optimization details` +- ``.gsd/milestones/M025/slices/S10/S10-SUMMARY.md` — requirement validation details` +- ``.gsd/milestones/M025/slices/S11/S11-RESEARCH.md` — wiki page inventory and update plan` + +## Expected Output + +- ``Home.md` — updated with M025 feature summary on ub01 wiki` +- ``API-Surface.md` — updated with 6+ new endpoints` +- ``Data-Model.md` — updated with EmailDigestLog, ChatUsageLog, new User columns` +- ``Frontend.md` — updated with 4 new pages/components` +- ``Configuration.md` — updated with SMTP, rate limit, fallback env vars` +- ``Chat-Engine.md` — updated with fallback resilience and prompt optimization` +- ``Deployment.md` — updated with Celery Beat and fallback config` +- ``Decisions.md` — appended D045, D046` +- ``_Sidebar.md` — updated with Newcomer Guide link` +- ``Architecture.md` — updated with email digest async subsystem` + +## Verification + +ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && git log --oneline -1' shows M025 commit AND ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && git diff HEAD~1 --stat' shows ~10 files changed diff --git a/.gsd/milestones/M025/slices/S11/tasks/T01-SUMMARY.md b/.gsd/milestones/M025/slices/S11/tasks/T01-SUMMARY.md new file mode 100644 index 0000000..41148fa --- /dev/null +++ b/.gsd/milestones/M025/slices/S11/tasks/T01-SUMMARY.md @@ -0,0 +1,93 @@ +--- +id: T01 +parent: S11 +milestone: M025 +provides: [] +requires: [] +affects: [] +key_files: ["Home.md", "API-Surface.md", "Data-Model.md", "Frontend.md", "Configuration.md", "Chat-Engine.md", "Deployment.md", "Decisions.md", "_Sidebar.md", "Architecture.md"] +key_decisions: ["Added D047 (fail-open rate limiter) and D048 (LLM fallback catch-and-retry) alongside planned D045/D046"] +patterns_established: [] +drill_down_paths: [] +observability_surfaces: [] +duration: "" +verification_result: "git log shows M025 commit (ba72230), git diff HEAD~1 --stat shows exactly 10 files changed with 229 insertions. Content spot-checks confirmed all expected sections present in each file." +completed_at: 2026-04-04T15:32:39.290Z +blocker_discovered: false +--- + +# T01: Updated 10 Forgejo wiki pages with M025 feature documentation — notifications, rate limiting, onboarding, transparency, export, fallback, prompt optimization, mobile, quality toolkit + +> Updated 10 Forgejo wiki pages with M025 feature documentation — notifications, rate limiting, onboarding, transparency, export, fallback, prompt optimization, mobile, quality toolkit + +## What Happened +--- +id: T01 +parent: S11 +milestone: M025 +key_files: + - Home.md + - API-Surface.md + - Data-Model.md + - Frontend.md + - Configuration.md + - Chat-Engine.md + - Deployment.md + - Decisions.md + - _Sidebar.md + - Architecture.md +key_decisions: + - Added D047 (fail-open rate limiter) and D048 (LLM fallback catch-and-retry) alongside planned D045/D046 +duration: "" +verification_result: passed +completed_at: 2026-04-04T15:32:39.291Z +blocker_discovered: false +--- + +# T01: Updated 10 Forgejo wiki pages with M025 feature documentation — notifications, rate limiting, onboarding, transparency, export, fallback, prompt optimization, mobile, quality toolkit + +**Updated 10 Forgejo wiki pages with M025 feature documentation — notifications, rate limiting, onboarding, transparency, export, fallback, prompt optimization, mobile, quality toolkit** + +## What Happened + +Cloned wiki repo to /tmp/chrysopedia-wiki-m025 on ub01, wrote a Python update script that modified all 10 target pages: Home.md (M025 feature sections, updated counts), API-Surface.md (7 new endpoints across 4 sections), Data-Model.md (EmailDigestLog, ChatUsageLog, User column changes), Frontend.md (5 new pages/components), Configuration.md (SMTP, rate limit, LLM fallback vars), Chat-Engine.md (fallback resilience, refined prompt, quality toolkit), Deployment.md (Celery Beat, fallback config), Decisions.md (D045-D048), _Sidebar.md (Newcomer Guide link), Architecture.md (notification data flow, Redis description update). Committed locally — push happens in T02 alongside the Newcomer Guide. + +## Verification + +git log shows M025 commit (ba72230), git diff HEAD~1 --stat shows exactly 10 files changed with 229 insertions. Content spot-checks confirmed all expected sections present in each file. + +## Verification Evidence + +| # | Command | Exit Code | Verdict | Duration | +|---|---------|-----------|---------|----------| +| 1 | `ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && git log --oneline -1'` | 0 | ✅ pass — shows M025 commit | 1000ms | +| 2 | `ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && git diff HEAD~1 --stat'` | 0 | ✅ pass — 10 files changed, 229 insertions | 1000ms | + + +## Deviations + +Added D047 and D048 decisions alongside planned D045/D046 — these were meaningful decisions from M025 that deserved wiki documentation. + +## Known Issues + +None. + +## Files Created/Modified + +- `Home.md` +- `API-Surface.md` +- `Data-Model.md` +- `Frontend.md` +- `Configuration.md` +- `Chat-Engine.md` +- `Deployment.md` +- `Decisions.md` +- `_Sidebar.md` +- `Architecture.md` + + +## Deviations +Added D047 and D048 decisions alongside planned D045/D046 — these were meaningful decisions from M025 that deserved wiki documentation. + +## Known Issues +None. diff --git a/.gsd/milestones/M025/slices/S11/tasks/T02-PLAN.md b/.gsd/milestones/M025/slices/S11/tasks/T02-PLAN.md new file mode 100644 index 0000000..57145ae --- /dev/null +++ b/.gsd/milestones/M025/slices/S11/tasks/T02-PLAN.md @@ -0,0 +1,95 @@ +--- +estimated_steps: 60 +estimated_files: 1 +skills_used: [] +--- + +# T02: Create Newcomer Guide and push all wiki changes + +Create the Newcomer-Guide.md page — the primary deliverable of this slice — and push all wiki changes (both T01 updates and this new page) to the Forgejo wiki. + +**CRITICAL:** Never use the Forgejo wiki PATCH API — it corrupts pages. Only use git push. + +**Wiki clone location:** `/tmp/chrysopedia-wiki-m025` on ub01 (set up by T01). + +**Newcomer-Guide.md structure (≥8 sections):** + +1. **What is Chrysopedia** — 3-sentence pitch: AI-powered knowledge base for music production techniques, built from video content by creators. Extracts, structures, and makes searchable the knowledge embedded in production tutorials and livestreams. + +2. **How Content Flows** — Simplified pipeline overview: + - Video transcripts land in a watched folder + - 6-stage pipeline: transcript segmentation → key moment extraction → classification/tagging → technique page synthesis → embedding/indexing → (optional) highlight detection + - Results: structured technique pages, searchable key moments, cross-references between creators + +3. **Using the Web UI** — End-user features: + - Search: typeahead search from homepage or Cmd+K anywhere, semantic + keyword + - Browse Topics: 7 top-level categories → sub-topics → technique pages grouped by creator + - Browse Creators: filterable list with genre tags, randomized default sort + - Technique Pages: study guide prose, key moments index, related techniques, table of contents + - Chat: creator-scoped AI chat with citation, cascade retrieval (creator→domain→global) + - Reading Header: sticky section bar on technique pages when scrolling + +4. **For Creators** — Creator-specific features: + - Registration and onboarding wizard (3-step: welcome, consent, tour) + - Consent dashboard: control which content types are published + - Creator dashboard: overview stats, recent posts, technique pages + - Transparency page: view all entities, relationships, technique pages derived from content + - Data export: GDPR-style ZIP download of all derived content + - Notification preferences: email digest frequency control + - Personality profiles: 5-tier system for chat persona customization + +5. **For Admins** — Administrative features: + - Review queue: approve/reject/edit key moments by source video + - Pipeline admin: monitor processing stages, filter by creator/status + - Usage dashboard: token consumption tracking, top creators/users, daily stats + - Audit log: all administrative actions tracked + - User management: role assignment, account status + +6. **Adding New Content** — Content ingestion workflow: + - Prepare video transcripts (Whisper large-v3 on RTX 4090) + - Place transcript JSON in watched folder + - Pipeline processes automatically through all stages + - Review queue for quality control before publishing + +7. **Infrastructure & Deployment** — High-level deployment info: + - Docker Compose stack on ub01 + - Services: API, Worker (Celery + Beat), Web (nginx), PostgreSQL, Redis, Qdrant, Ollama, LightRAG + - LLM: primary DGX endpoint with automatic Ollama fallback + - Link to Deployment and Development-Guide wiki pages + +8. **Where to Learn More** — Cross-reference index linking to every wiki page by topic: + - Architecture overview → Architecture + - API reference → API-Surface + - Data model → Data-Model + - Search & retrieval → Search-Retrieval + - Chat engine → Chat-Engine + - Pipeline stages → Pipeline + - Authentication → Authentication + - Configuration → Configuration + - Monitoring → Monitoring + - etc. + +**After creating the file, push all changes:** +```bash +ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && git add . && git commit -m "docs: M025 wiki update — notifications, rate limiting, onboarding, transparency, export, fallback, mobile, newcomer guide" && git push' +``` + +**Verify page count via API:** +```bash +ssh ub01 'curl -s "https://git.xpltd.co/api/v1/repos/xpltdco/chrysopedia/wiki/pages?limit=50" | python3 -c "import json,sys; pages=json.load(sys.stdin); print(len(pages), chr(39) + chr(39).join([]) + \"pages\")"' +``` +Expect ≥21 pages. + +## Inputs + +- ``.gsd/milestones/M025/slices/S11/S11-RESEARCH.md` — newcomer guide structure recommendation` +- ``Home.md` — T01 output, for cross-reference consistency` +- ``_Sidebar.md` — T01 output, link already added` + +## Expected Output + +- ``Newcomer-Guide.md` — new wiki page with ≥8 sections covering the full platform` + +## Verification + +ssh ub01 'curl -s "https://git.xpltd.co/api/v1/repos/xpltdco/chrysopedia/wiki/pages?limit=50" | python3 -c "import json,sys; pages=json.load(sys.stdin); print(len(pages))"' returns ≥21 AND ssh ub01 'cd /tmp/chrysopedia-wiki-m025 && test -f Newcomer-Guide.md && grep -c "^## " Newcomer-Guide.md' returns ≥8