---
id: M007
title: "Pipeline Transparency, Auto-Ingest, Admin UX Polish, and Mobile Fixes"
status: complete
completed_at: 2026-03-30T19:52:33.256Z
key_decisions:
  - Followed Redis-backed toggle pattern (from review_mode) for debug_mode — consistent admin toggle approach across the codebase
  - Debug mode checked once at callback creation time rather than per-LLM-call to avoid Redis round-trip overhead
  - Full response text stored without truncation when debug on; existing content_preview kept for non-debug use
  - Used watchdog PollingObserver over inotify for ZFS/NFS reliability on ub01
  - Used os.kill(1,0) healthcheck for watcher container (procps unavailable in slim Python image)
  - Applied flex-wrap at base CSS level rather than only in mobile media query — wrapping works at all widths
  - Used word-break: break-word for titles (preserves readability) vs text-overflow: ellipsis for filenames (less meaningful content)
key_files:
  - backend/pipeline/stages.py
  - backend/watcher.py
  - backend/routers/pipeline.py
  - backend/schemas.py
  - backend/models.py
  - backend/config.py
  - alembic/versions/006_debug_columns.py
  - frontend/src/pages/AdminPipeline.tsx
  - frontend/src/api/public-client.ts
  - frontend/src/App.css
  - docker-compose.yml
  - backend/requirements.txt
lessons_learned:
  - PollingObserver is more reliable than inotify Observer on ZFS/NFS filesystems — inotify doesn't reliably detect changes on network or ZFS mounts
  - procps (pgrep) is not available in slim Python Docker images — use os.kill(pid, 0) as an alternative process-existence healthcheck
  - File stability detection (checking size hasn't changed over N seconds) is essential for folder watchers that may see partial SCP/rsync writes
  - When debug/diagnostic data is stored conditionally (toggle-based), check the toggle once at initialization rather than per-operation to minimize overhead
---

# M007: Pipeline Transparency, Auto-Ingest, Admin UX Polish, and Mobile Fixes

**Made the LLM pipeline fully transparent with debug mode I/O capture and token accounting, automated transcript ingestion via folder watcher, streamlined the admin UX, and fixed frontend overflow bugs on mobile viewports.**

## What Happened

M007 delivered six slices across three themes: pipeline observability, workflow automation, and frontend polish.

**Pipeline transparency (S01 + S02):** Added a Redis-backed debug mode toggle that captures full system prompt, user prompt, and response text for every LLM call in pipeline_events. Per-stage token summary endpoint provides cost/usage breakdowns. The admin UI got a DebugPayloadViewer component — collapsible sections for each prompt/response, per-section clipboard copy, and JSON export. This gives complete visibility into what the LLM sees and produces at every stage.

**Transcript auto-ingest (S03):** Built and deployed chrysopedia-watcher, a new Docker service using watchdog's PollingObserver to monitor /vmPool/r/services/chrysopedia_watch/ on ub01. Drop a transcript JSON in the folder → file stability check → JSON validation → POST to /ingest → file moved to processed/ or failed/ with error sidecar. Eliminates the manual curl workflow for pipeline input.

**Admin UX audit (S04):** Added DebugModeToggle in the header, StatusFilter pill bar with per-status counts, renamed cryptic Head/Tail toggle to Oldest/Newest first, removed duplicate refresh button and low-value UUID display, added debug indicator on trigger button, and cross-link to review queue per video.

**Frontend overflow fixes (S05 + S06):** Fixed key moment card text overflow with word-break, max-width constraints, and flex child min-width. Fixed technique page mobile overflow with flex-wrap on tags and genre pills, max-width on version switcher, and tighter padding/gaps at ≤640px breakpoints.

## Success Criteria Results

The roadmap did not define explicit success criteria — the milestone vision serves as the implicit criteria:

- ✅ **Every LLM call's full prompt and response viewable, copyable, exportable** — S01 stores system_prompt_text, user_prompt_text, response_text in pipeline_events when debug mode is on. S02 renders these in a collapsible viewer with copy and JSON export.
- ✅ **Per-stage token accounting** — S01 added GET /api/v1/admin/pipeline/token-summary/{video_id} returning per-stage call counts and token breakdowns with grand total.
- ✅ **Automate transcript ingestion via folder monitoring** — S03 deployed chrysopedia-watcher service. End-to-end verified: valid JSON auto-ingested (HTTP 200), invalid JSON moved to failed/ with .error sidecar.
- ✅ **Tighten admin UX: prune dead weight, improve workflow clarity** — S04 added debug toggle, status filter, renamed labels, removed dead UI, added cross-links.
- ✅ **Fix frontend overflow and mobile responsiveness bugs** — S05 fixed key moment card overflow. S06 fixed technique page mobile overflow at ~412px viewports.

## Definition of Done Results

- ✅ All 6 slices complete with ✅ in roadmap
- ✅ All 6 slice summaries exist (S01 through S06)
- ✅ Cross-slice integration: S01→S02 (debug data captured → viewer renders it), S02→S04 (viewer present → UX audit builds on it), S05→S06 (card overflow fixed → mobile viewport fix builds on stable cards)
- ✅ Code changes verified: 14 files, 1,341 insertions across backend, frontend, and Docker config
- ✅ All services deployed and running on ub01

## Requirement Outcomes

- **R002 (Transcript Ingestion API):** Remains validated. Advanced by S03 — ingestion now automated via folder watcher, eliminating manual curl/upload workflow.
- **R006 (Technique Page Display):** Remains validated. Advanced by S06 — technique page now renders cleanly on mobile viewports (tags wrap, metadata stays within bounds, no horizontal scroll).

## Deviations

Docker service naming inconsistency: plan documents referenced chrysopedia-web-8096 but the Docker Compose build target is chrysopedia-web (the -8096 suffix is only the container name). Minor, no functional impact. S02 changes were applied directly on ub01 rather than through git push due to diverged branches — all code lives on ub01 canonical copy.

## Follow-ups

Add URL-based video filtering to the review queue so the → Moments cross-link from AdminPipeline can deep-link to a specific video's moments. Consider adding per-video debug mode (currently global toggle). Add automated visual regression testing for mobile layouts.
