feat: Added 3 TEXT columns to pipeline_events (system_prompt_text, user…
- "backend/models.py" - "backend/config.py" - "backend/schemas.py" - "backend/routers/pipeline.py" - "alembic/versions/006_debug_columns.py" GSD-Task: S01/T01
This commit is contained in:
jlightner
parent
0484c15516
commit
be6b2d5963
12 changed files with 529 additions and 0 deletions
14
.gsd/milestones/M007/M007-ROADMAP.md
Normal file
14
.gsd/milestones/M007/M007-ROADMAP.md
Normal file
|
|
@ -0,0 +1,14 @@
|
||||||
|
# M007:
|
||||||
|
|
||||||
|
## Vision
|
||||||
|
Make the pipeline fully transparent — every LLM call's full prompt and response viewable, copyable, exportable, with per-stage token accounting. Automate transcript ingestion via folder monitoring instead of manual POST. Tighten the admin UX: prune dead weight, improve workflow clarity. Fix frontend overflow and mobile responsiveness bugs. Set the foundation for iterating on content quality through better visibility into what the LLM is actually doing.
|
||||||
|
|
||||||
|
## Slice Overview
|
||||||
|
| ID | Slice | Risk | Depends | Done | After this |
|
||||||
|
|----|-------|------|---------|------|------------|
|
||||||
|
| S01 | Pipeline Debug Mode — Full LLM I/O Capture and Token Accounting | medium | — | ⬜ | Toggle debug mode on for a video, trigger pipeline, see full prompt + response stored for every LLM call with per-stage token breakdown |
|
||||||
|
| S02 | Debug Payload Viewer — Inline View, Copy, and Export in Admin UI | low | S01 | ⬜ | Admin clicks an LLM call event, sees full prompt and response in a readable viewer, copies to clipboard, or downloads as JSON |
|
||||||
|
| S03 | Transcript Folder Watcher — Auto-Ingest Service | medium | — | ⬜ | Drop a transcript JSON into the watch folder on ub01 → file is detected, validated, POSTed to /ingest, pipeline triggers automatically |
|
||||||
|
| S04 | Admin UX Audit — Prune, Streamline, and Polish | low | S02 | ⬜ | Admin pipeline page is cleaner and more efficient for daily content management. Dead UI elements removed. Workflow improvements visible. |
|
||||||
|
| S05 | Key Moment Card Text Overflow Fix | low | — | ⬜ | Key moment cards with long filenames and timestamp ranges display cleanly — text truncated with ellipsis or wrapped within card bounds, no horizontal bleed |
|
||||||
|
| S06 | Mobile Viewport Overflow Fix — Technique Pages and Global Content | low | S05 | ⬜ | Technique page on Galaxy S25 Ultra renders cleanly — body text wraps, tags don't overflow, metadata stays within viewport, no horizontal scroll |
|
||||||
123
.gsd/milestones/M007/slices/S01/S01-PLAN.md
Normal file
123
.gsd/milestones/M007/slices/S01/S01-PLAN.md
Normal file
|
|
@ -0,0 +1,123 @@
|
||||||
|
# S01: Pipeline Debug Mode — Full LLM I/O Capture and Token Accounting
|
||||||
|
|
||||||
|
**Goal:** Debug mode toggle (Redis-backed) controls full LLM I/O capture in the pipeline. When on, every LLM call stores system prompt, user prompt, and full response text. Per-stage token summary available via API.
|
||||||
|
**Demo:** After this: Toggle debug mode on for a video, trigger pipeline, see full prompt + response stored for every LLM call with per-stage token breakdown
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
- [x] **T01: Added 3 TEXT columns to pipeline_events (system_prompt_text, user_prompt_text, response_text), Redis-backed debug mode toggle, and per-stage token summary endpoint** — Set up all infrastructure for debug mode: Alembic migration adding 3 TEXT columns to pipeline_events, PipelineEvent model update, debug mode GET/PUT endpoints (Redis-backed, following review_mode pattern from D007), token summary aggregation endpoint, and extended event listing response.
|
||||||
|
|
||||||
|
All work is on ub01 via SSH at /vmPool/r/repos/xpltdco/chrysopedia.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Add `system_prompt_text`, `user_prompt_text`, `response_text` columns (all `Text, nullable=True`) to `PipelineEvent` model in `backend/models.py` after the existing `payload` column.
|
||||||
|
|
||||||
|
2. Create Alembic migration `alembic/versions/006_debug_columns.py` that adds the 3 TEXT columns to `pipeline_events`. Use `op.add_column()` with `nullable=True` — no data rewrite needed.
|
||||||
|
|
||||||
|
3. Add `debug_mode: bool = False` to `Settings` class in `backend/config.py`.
|
||||||
|
|
||||||
|
4. Add Pydantic schemas in `backend/schemas.py`:
|
||||||
|
- `DebugModeResponse(BaseModel)` with `debug_mode: bool`
|
||||||
|
- `DebugModeUpdate(BaseModel)` with `debug_mode: bool`
|
||||||
|
- `TokenStageSummary(BaseModel)` with `stage: str`, `call_count: int`, `total_prompt_tokens: int`, `total_completion_tokens: int`, `total_tokens: int`
|
||||||
|
- `TokenSummaryResponse(BaseModel)` with `video_id: str`, `stages: list[TokenStageSummary]`, `grand_total_tokens: int`
|
||||||
|
|
||||||
|
5. Add 3 new endpoints in `backend/routers/pipeline.py`:
|
||||||
|
- `GET /admin/pipeline/debug-mode` — read from Redis key `chrysopedia:debug_mode`, fall back to `settings.debug_mode`. Follow exact pattern from review router's `get_mode()`.
|
||||||
|
- `PUT /admin/pipeline/debug-mode` — write to Redis. Follow exact pattern from review router's `set_mode()`.
|
||||||
|
- `GET /admin/pipeline/token-summary/{video_id}` — query `pipeline_events` grouped by stage where `event_type='llm_call'`, sum token columns, return `TokenSummaryResponse`.
|
||||||
|
|
||||||
|
6. Extend the existing `list_pipeline_events` response dict to include `system_prompt_text`, `user_prompt_text`, `response_text` fields (nullable).
|
||||||
|
|
||||||
|
7. Run migration: `docker exec chrysopedia-api alembic upgrade head`. Rebuild API container if needed: `docker compose build chrysopedia-api && docker compose up -d chrysopedia-api`.
|
||||||
|
|
||||||
|
8. Verify: curl debug-mode GET returns `{"debug_mode": false}`, PUT sets to true, GET reads back true. Curl token-summary for any video_id returns valid (possibly empty) response.
|
||||||
|
|
||||||
|
## Must-Haves
|
||||||
|
|
||||||
|
- [ ] 3 TEXT columns added to pipeline_events table via migration
|
||||||
|
- [ ] PipelineEvent model has matching mapped columns
|
||||||
|
- [ ] Debug mode GET/PUT endpoints work with Redis (fall back to config)
|
||||||
|
- [ ] Token summary endpoint returns per-stage aggregation
|
||||||
|
- [ ] Event listing includes new fields
|
||||||
|
|
||||||
|
## Failure Modes
|
||||||
|
|
||||||
|
| Dependency | On error | On timeout | On malformed response |
|
||||||
|
|------------|----------|-----------|----------------------|
|
||||||
|
| Redis (debug mode read) | Fall back to config.debug_mode default | Same as error — use default | N/A (simple GET/SET of string) |
|
||||||
|
| PostgreSQL (token summary) | Return 500 with error detail | Standard DB timeout handling | N/A (aggregation query) |
|
||||||
|
- Estimate: 1.5h
|
||||||
|
- Files: backend/models.py, backend/config.py, backend/schemas.py, backend/routers/pipeline.py, alembic/versions/006_debug_columns.py
|
||||||
|
- Verify: ssh ub01 'docker exec chrysopedia-api alembic upgrade head' && curl -sf http://ub01:8096/admin/pipeline/debug-mode | grep debug_mode
|
||||||
|
- [ ] **T02: Wire debug-mode-aware LLM I/O capture into pipeline callback and verify end-to-end** — Modify the pipeline's `_make_llm_callback` to conditionally capture full prompt and response text when debug mode is enabled. Thread system_prompt and user_prompt through to the callback at all 4 stage call sites. Rebuild worker container and verify end-to-end on ub01.
|
||||||
|
|
||||||
|
All work is on ub01 via SSH at /vmPool/r/repos/xpltdco/chrysopedia.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Add `system_prompt_text`, `user_prompt_text`, `response_text` kwargs to `_emit_event()` in `backend/pipeline/stages.py`. Pass them through to the `PipelineEvent` constructor.
|
||||||
|
|
||||||
|
2. Add a sync Redis helper at the top of `backend/pipeline/stages.py` to read debug mode:
|
||||||
|
```python
|
||||||
|
def _is_debug_mode() -> bool:
|
||||||
|
"""Check if debug mode is enabled via Redis. Falls back to config."""
|
||||||
|
try:
|
||||||
|
import redis
|
||||||
|
settings = get_settings()
|
||||||
|
r = redis.from_url(settings.redis_url)
|
||||||
|
val = r.get("chrysopedia:debug_mode")
|
||||||
|
r.close()
|
||||||
|
if val is not None:
|
||||||
|
return val.decode().lower() == "true"
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return getattr(get_settings(), 'debug_mode', False)
|
||||||
|
```
|
||||||
|
Note: uses sync `redis.from_url()` because Celery worker is sync (D004). The `redis` package is already installed (Celery broker dependency).
|
||||||
|
|
||||||
|
3. Modify `_make_llm_callback(video_id, stage)` to accept `system_prompt: str | None = None` and `user_prompt: str | None = None` parameters. Inside the callback closure:
|
||||||
|
- Check `_is_debug_mode()`
|
||||||
|
- If debug on: pass `system_prompt_text=system_prompt`, `user_prompt_text=user_prompt`, `response_text=content` to `_emit_event()`
|
||||||
|
- If debug off: pass None for all three (existing behavior preserved)
|
||||||
|
|
||||||
|
4. Update all 4 `llm.complete()` call sites to pass prompt text to the callback factory:
|
||||||
|
- Stage 2 (~line 278): `_make_llm_callback(video_id, "stage2_segmentation", system_prompt=system_prompt, user_prompt=user_prompt)`
|
||||||
|
- Stage 3 (~line 370): same pattern
|
||||||
|
- Stage 4 (~line 485): same pattern
|
||||||
|
- Stage 5 (~line 724): same pattern
|
||||||
|
|
||||||
|
5. Rebuild and restart worker + API containers:
|
||||||
|
```bash
|
||||||
|
cd /vmPool/r/repos/xpltdco/chrysopedia
|
||||||
|
docker compose build chrysopedia-api chrysopedia-worker
|
||||||
|
docker compose up -d
|
||||||
|
docker compose restart chrysopedia-web-8096 # nginx DNS cache
|
||||||
|
```
|
||||||
|
|
||||||
|
6. End-to-end verification:
|
||||||
|
a. Enable debug mode: `curl -X PUT http://ub01:8096/admin/pipeline/debug-mode -H 'Content-Type: application/json' -d '{"debug_mode":true}'`
|
||||||
|
b. Trigger pipeline for a video: `curl -X POST http://ub01:8096/admin/pipeline/trigger/{video_id}`
|
||||||
|
c. Wait for pipeline completion (check logs or events)
|
||||||
|
d. Query events: `curl http://ub01:8096/admin/pipeline/events/{video_id}?event_type=llm_call` — verify `system_prompt_text`, `user_prompt_text`, `response_text` are populated
|
||||||
|
e. Query token summary: `curl http://ub01:8096/admin/pipeline/token-summary/{video_id}` — verify per-stage breakdown
|
||||||
|
f. Disable debug mode: `curl -X PUT http://ub01:8096/admin/pipeline/debug-mode -H 'Content-Type: application/json' -d '{"debug_mode":false}'`
|
||||||
|
g. (Optional) Trigger another run — verify new events have null prompt/response columns
|
||||||
|
|
||||||
|
## Must-Haves
|
||||||
|
|
||||||
|
- [ ] `_emit_event` accepts and stores prompt/response text columns
|
||||||
|
- [ ] `_is_debug_mode()` reads from sync Redis, falls back to config
|
||||||
|
- [ ] `_make_llm_callback` conditionally captures full I/O when debug is on
|
||||||
|
- [ ] All 4 stage call sites pass system_prompt and user_prompt to callback
|
||||||
|
- [ ] Pipeline still works when debug is off (existing behavior preserved)
|
||||||
|
- [ ] End-to-end: debug on → pipeline → events have full prompt/response text
|
||||||
|
|
||||||
|
## Observability Impact
|
||||||
|
|
||||||
|
- Signals added: Full LLM system prompt, user prompt, and response text stored per llm_call event when debug mode is on
|
||||||
|
- How a future agent inspects this: `curl http://ub01:8096/admin/pipeline/events/{video_id}?event_type=llm_call` — check system_prompt_text/user_prompt_text/response_text fields
|
||||||
|
- Failure state exposed: `_is_debug_mode()` failure is silent (falls back to off) — matches existing best-effort pattern in `_emit_event`
|
||||||
|
- Estimate: 1.5h
|
||||||
|
- Files: backend/pipeline/stages.py
|
||||||
|
- Verify: ssh ub01 'curl -sf http://ub01:8096/admin/pipeline/debug-mode | python3 -c "import sys,json; assert json.load(sys.stdin)[\"debug_mode\"] in [True,False]"' && ssh ub01 'docker logs chrysopedia-worker --tail 5 2>&1 | head -5'
|
||||||
85
.gsd/milestones/M007/slices/S01/S01-RESEARCH.md
Normal file
85
.gsd/milestones/M007/slices/S01/S01-RESEARCH.md
Normal file
|
|
@ -0,0 +1,85 @@
|
||||||
|
# S01 Research — Pipeline Debug Mode: Full LLM I/O Capture and Token Accounting
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
The existing `PipelineEvent` model and `_make_llm_callback` infrastructure captures token counts and a truncated 2000-char `content_preview` for each LLM call. **No prompt text is stored.** Full response text is discarded. This slice needs to:
|
||||||
|
|
||||||
|
1. Add a debug mode toggle (global, Redis-backed like `review_mode`)
|
||||||
|
2. When debug is on, capture full system prompt, user prompt, and full response text for every LLM call
|
||||||
|
3. Expose per-stage token aggregation via API
|
||||||
|
|
||||||
|
The work is straightforward — extending an established event-capture pattern with two new columns, a toggle, and an aggregation query.
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
**Targeted approach — extend existing infrastructure, don't rebuild it.**
|
||||||
|
|
||||||
|
- Add `system_prompt_text` and `user_prompt_text` TEXT columns to `pipeline_events` (alongside existing `payload` JSONB which keeps the response content). Alternatively, store full response in a dedicated `response_text` TEXT column for symmetry and queryability, keeping `payload` for metadata only.
|
||||||
|
- Debug mode: global Redis key `chrysopedia:debug_mode` (follows `review_mode` pattern from D007). Config default `debug_mode: bool = False`. When debug is **off**, behavior is unchanged (truncated preview only). When **on**, full prompts + full response stored.
|
||||||
|
- Thread prompt text into the callback by expanding `_make_llm_callback` to accept `system_prompt` and `user_prompt` parameters. The callback closure already captures `video_id` and `stage`.
|
||||||
|
- Token aggregation: add `GET /admin/pipeline/token-summary/{video_id}` that groups `pipeline_events` by `stage` and sums token columns.
|
||||||
|
|
||||||
|
## Implementation Landscape
|
||||||
|
|
||||||
|
### Existing Code (what to modify)
|
||||||
|
|
||||||
|
| File | Role | What changes |
|
||||||
|
|------|------|-------------|
|
||||||
|
| `backend/models.py` (line ~386) | `PipelineEvent` model | Add `system_prompt_text`, `user_prompt_text`, `response_text` — all `Text, nullable=True` |
|
||||||
|
| `backend/pipeline/stages.py` (line ~52) | `_emit_event()` | Accept new `system_prompt_text`, `user_prompt_text`, `response_text` kwargs, pass to model |
|
||||||
|
| `backend/pipeline/stages.py` (line ~87) | `_make_llm_callback()` | Accept `system_prompt`, `user_prompt` params; conditionally capture full text when debug mode is on |
|
||||||
|
| `backend/pipeline/stages.py` (lines 275-280, 365-372, 478-487, 720-726) | Stage `llm.complete()` call sites | Pass `system_prompt` and `user_prompt` to `_make_llm_callback()` |
|
||||||
|
| `backend/config.py` | `Settings` | Add `debug_mode: bool = False` |
|
||||||
|
| `backend/routers/pipeline.py` | Pipeline admin endpoints | Add debug mode GET/PUT, token summary endpoint, include new fields in event response |
|
||||||
|
| `backend/schemas.py` | Pydantic schemas | Add `DebugModeResponse`, `DebugModeUpdate`, `TokenSummary` schemas |
|
||||||
|
| `alembic/versions/006_debug_columns.py` | New migration | Add 3 TEXT columns to `pipeline_events` |
|
||||||
|
|
||||||
|
### Key Design Details
|
||||||
|
|
||||||
|
**Storage sizing:** System prompts are 4-12KB. User prompts include transcript text — can be 50-200KB for long videos. Full LLM responses are similar magnitude to user prompts. At ~4 LLM calls per video (stages 2-5, though stage 3 and 5 may loop), worst case is ~1-2MB per video pipeline run. PostgreSQL TEXT columns handle this fine. No need for filesystem or blob storage at this scale.
|
||||||
|
|
||||||
|
**Stage 3 and 5 looping:** Stage 3 calls LLM once per topic group; stage 5 calls once per (creator, category) group. Each call already produces a separate `PipelineEvent` with `event_type="llm_call"`. The existing infrastructure handles multi-call stages correctly — no special grouping needed.
|
||||||
|
|
||||||
|
**Debug mode check location:** The `_make_llm_callback` factory runs in the Celery worker (sync context). It needs to read debug mode. Two options:
|
||||||
|
- Read from config `settings.debug_mode` (simpler, requires worker restart to change)
|
||||||
|
- Read from Redis via sync `redis.Redis` (matches `review_mode` runtime toggle pattern, no restart needed)
|
||||||
|
|
||||||
|
**Recommendation:** Use sync Redis read in the callback. The worker already imports `config.get_settings()` which has `redis_url`. Create a thin sync Redis helper (or just `redis.from_url()` inline) — the async `redis_client.py` helper is for FastAPI only.
|
||||||
|
|
||||||
|
**`_safe_parse_llm_response` retry calls:** When the first LLM parse fails, `_safe_parse_llm_response` makes a retry call with a nudge prompt. This retry call does NOT have an `on_complete` callback — it won't emit a debug event. This is acceptable for now (retry is a diagnostic edge case), but note it for S02 if full call tracing is needed.
|
||||||
|
|
||||||
|
**Per-stage token aggregation query:**
|
||||||
|
```sql
|
||||||
|
SELECT stage,
|
||||||
|
COUNT(*) AS call_count,
|
||||||
|
SUM(prompt_tokens) AS total_prompt_tokens,
|
||||||
|
SUM(completion_tokens) AS total_completion_tokens,
|
||||||
|
SUM(total_tokens) AS total_tokens
|
||||||
|
FROM pipeline_events
|
||||||
|
WHERE video_id = :vid AND event_type = 'llm_call'
|
||||||
|
GROUP BY stage
|
||||||
|
ORDER BY MIN(created_at)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Natural Task Seams
|
||||||
|
|
||||||
|
1. **DB migration + model** — Add 3 TEXT columns to `pipeline_events`, update `PipelineEvent` model. Pure schema work, verifiable with `alembic upgrade head`.
|
||||||
|
2. **Debug mode toggle** — Config setting + Redis key + GET/PUT endpoints + schemas. Follows `review_mode` pattern exactly. Verifiable via API calls.
|
||||||
|
3. **Capture instrumentation** — Modify `_make_llm_callback` to accept and conditionally store prompts/response. Modify all 4 stage call sites to pass prompt text. This is the core work. Verifiable by triggering pipeline with debug on and querying events.
|
||||||
|
4. **Token summary endpoint** — New `GET /admin/pipeline/token-summary/{video_id}` with aggregation query. Verifiable via API call after pipeline run.
|
||||||
|
|
||||||
|
Tasks 1-2 are independent. Task 3 depends on 1 (columns must exist). Task 4 depends on nothing new (uses existing columns).
|
||||||
|
|
||||||
|
### Constraints & Risks
|
||||||
|
|
||||||
|
- **Celery worker sync context:** The debug mode Redis read must use sync `redis.Redis`, not the async client from `redis_client.py`. This is the same constraint that led to sync SQLAlchemy in the worker (D004).
|
||||||
|
- **Large TEXT columns:** No index needed on prompt/response text. These are write-once, read-rarely columns for debugging. The existing `ix_pipeline_events_video_created` index handles all query patterns.
|
||||||
|
- **Migration on production data:** The `pipeline_events` table has existing rows. Adding nullable TEXT columns is a non-blocking ALTER TABLE in PostgreSQL — no data rewrite needed.
|
||||||
|
|
||||||
|
### Verification Strategy
|
||||||
|
|
||||||
|
1. Migration: `docker exec chrysopedia-api alembic upgrade head` succeeds, columns visible in `\d pipeline_events`
|
||||||
|
2. Debug toggle: `curl` GET/PUT on debug mode endpoint returns expected values
|
||||||
|
3. Full capture: Enable debug mode, trigger pipeline for a video, query events — `system_prompt_text`, `user_prompt_text`, `response_text` populated for `llm_call` events
|
||||||
|
4. Token summary: `GET /admin/pipeline/token-summary/{video_id}` returns per-stage breakdown
|
||||||
|
5. Debug off: Disable debug mode, trigger pipeline — new events have NULL prompt/response text columns (existing truncated `content_preview` in payload still works)
|
||||||
72
.gsd/milestones/M007/slices/S01/tasks/T01-PLAN.md
Normal file
72
.gsd/milestones/M007/slices/S01/tasks/T01-PLAN.md
Normal file
|
|
@ -0,0 +1,72 @@
|
||||||
|
---
|
||||||
|
estimated_steps: 29
|
||||||
|
estimated_files: 5
|
||||||
|
skills_used: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# T01: Add debug columns, migration, debug toggle endpoints, and token summary API
|
||||||
|
|
||||||
|
Set up all infrastructure for debug mode: Alembic migration adding 3 TEXT columns to pipeline_events, PipelineEvent model update, debug mode GET/PUT endpoints (Redis-backed, following review_mode pattern from D007), token summary aggregation endpoint, and extended event listing response.
|
||||||
|
|
||||||
|
All work is on ub01 via SSH at /vmPool/r/repos/xpltdco/chrysopedia.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Add `system_prompt_text`, `user_prompt_text`, `response_text` columns (all `Text, nullable=True`) to `PipelineEvent` model in `backend/models.py` after the existing `payload` column.
|
||||||
|
|
||||||
|
2. Create Alembic migration `alembic/versions/006_debug_columns.py` that adds the 3 TEXT columns to `pipeline_events`. Use `op.add_column()` with `nullable=True` — no data rewrite needed.
|
||||||
|
|
||||||
|
3. Add `debug_mode: bool = False` to `Settings` class in `backend/config.py`.
|
||||||
|
|
||||||
|
4. Add Pydantic schemas in `backend/schemas.py`:
|
||||||
|
- `DebugModeResponse(BaseModel)` with `debug_mode: bool`
|
||||||
|
- `DebugModeUpdate(BaseModel)` with `debug_mode: bool`
|
||||||
|
- `TokenStageSummary(BaseModel)` with `stage: str`, `call_count: int`, `total_prompt_tokens: int`, `total_completion_tokens: int`, `total_tokens: int`
|
||||||
|
- `TokenSummaryResponse(BaseModel)` with `video_id: str`, `stages: list[TokenStageSummary]`, `grand_total_tokens: int`
|
||||||
|
|
||||||
|
5. Add 3 new endpoints in `backend/routers/pipeline.py`:
|
||||||
|
- `GET /admin/pipeline/debug-mode` — read from Redis key `chrysopedia:debug_mode`, fall back to `settings.debug_mode`. Follow exact pattern from review router's `get_mode()`.
|
||||||
|
- `PUT /admin/pipeline/debug-mode` — write to Redis. Follow exact pattern from review router's `set_mode()`.
|
||||||
|
- `GET /admin/pipeline/token-summary/{video_id}` — query `pipeline_events` grouped by stage where `event_type='llm_call'`, sum token columns, return `TokenSummaryResponse`.
|
||||||
|
|
||||||
|
6. Extend the existing `list_pipeline_events` response dict to include `system_prompt_text`, `user_prompt_text`, `response_text` fields (nullable).
|
||||||
|
|
||||||
|
7. Run migration: `docker exec chrysopedia-api alembic upgrade head`. Rebuild API container if needed: `docker compose build chrysopedia-api && docker compose up -d chrysopedia-api`.
|
||||||
|
|
||||||
|
8. Verify: curl debug-mode GET returns `{"debug_mode": false}`, PUT sets to true, GET reads back true. Curl token-summary for any video_id returns valid (possibly empty) response.
|
||||||
|
|
||||||
|
## Must-Haves
|
||||||
|
|
||||||
|
- [ ] 3 TEXT columns added to pipeline_events table via migration
|
||||||
|
- [ ] PipelineEvent model has matching mapped columns
|
||||||
|
- [ ] Debug mode GET/PUT endpoints work with Redis (fall back to config)
|
||||||
|
- [ ] Token summary endpoint returns per-stage aggregation
|
||||||
|
- [ ] Event listing includes new fields
|
||||||
|
|
||||||
|
## Failure Modes
|
||||||
|
|
||||||
|
| Dependency | On error | On timeout | On malformed response |
|
||||||
|
|------------|----------|-----------|----------------------|
|
||||||
|
| Redis (debug mode read) | Fall back to config.debug_mode default | Same as error — use default | N/A (simple GET/SET of string) |
|
||||||
|
| PostgreSQL (token summary) | Return 500 with error detail | Standard DB timeout handling | N/A (aggregation query) |
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
- ``backend/models.py` — existing PipelineEvent model (line ~386)`
|
||||||
|
- ``backend/config.py` — existing Settings class with review_mode pattern`
|
||||||
|
- ``backend/schemas.py` — existing ReviewModeResponse/Update as template`
|
||||||
|
- ``backend/routers/pipeline.py` — existing pipeline endpoints and list_pipeline_events`
|
||||||
|
- ``backend/routers/review.py` — review_mode GET/PUT pattern to follow (lines 337-375)`
|
||||||
|
- ``alembic/versions/` — existing migrations for revision chain`
|
||||||
|
|
||||||
|
## Expected Output
|
||||||
|
|
||||||
|
- ``alembic/versions/006_debug_columns.py` — migration adding 3 TEXT columns`
|
||||||
|
- ``backend/models.py` — PipelineEvent with system_prompt_text, user_prompt_text, response_text columns`
|
||||||
|
- ``backend/config.py` — Settings with debug_mode field`
|
||||||
|
- ``backend/schemas.py` — DebugModeResponse, DebugModeUpdate, TokenStageSummary, TokenSummaryResponse`
|
||||||
|
- ``backend/routers/pipeline.py` — 3 new endpoints + extended event listing`
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
ssh ub01 'docker exec chrysopedia-api alembic upgrade head' && curl -sf http://ub01:8096/admin/pipeline/debug-mode | grep debug_mode
|
||||||
87
.gsd/milestones/M007/slices/S01/tasks/T01-SUMMARY.md
Normal file
87
.gsd/milestones/M007/slices/S01/tasks/T01-SUMMARY.md
Normal file
|
|
@ -0,0 +1,87 @@
|
||||||
|
---
|
||||||
|
id: T01
|
||||||
|
parent: S01
|
||||||
|
milestone: M007
|
||||||
|
provides: []
|
||||||
|
requires: []
|
||||||
|
affects: []
|
||||||
|
key_files: ["backend/models.py", "backend/config.py", "backend/schemas.py", "backend/routers/pipeline.py", "alembic/versions/006_debug_columns.py"]
|
||||||
|
key_decisions: ["Followed exact review_mode Redis pattern for debug_mode toggle", "Used coalesce in token aggregation to handle NULL token columns"]
|
||||||
|
patterns_established: []
|
||||||
|
drill_down_paths: []
|
||||||
|
observability_surfaces: []
|
||||||
|
duration: ""
|
||||||
|
verification_result: "Migration 006 applied successfully. Debug mode GET returns false (default), PUT toggles to true, GET reads back true. Token summary returns real per-stage aggregation for existing video data. DB columns confirmed via psql. Event listing response includes new nullable text fields."
|
||||||
|
completed_at: 2026-03-30T18:23:26.734Z
|
||||||
|
blocker_discovered: false
|
||||||
|
---
|
||||||
|
|
||||||
|
# T01: Added 3 TEXT columns to pipeline_events (system_prompt_text, user_prompt_text, response_text), Redis-backed debug mode toggle, and per-stage token summary endpoint
|
||||||
|
|
||||||
|
> Added 3 TEXT columns to pipeline_events (system_prompt_text, user_prompt_text, response_text), Redis-backed debug mode toggle, and per-stage token summary endpoint
|
||||||
|
|
||||||
|
## What Happened
|
||||||
|
---
|
||||||
|
id: T01
|
||||||
|
parent: S01
|
||||||
|
milestone: M007
|
||||||
|
key_files:
|
||||||
|
- backend/models.py
|
||||||
|
- backend/config.py
|
||||||
|
- backend/schemas.py
|
||||||
|
- backend/routers/pipeline.py
|
||||||
|
- alembic/versions/006_debug_columns.py
|
||||||
|
key_decisions:
|
||||||
|
- Followed exact review_mode Redis pattern for debug_mode toggle
|
||||||
|
- Used coalesce in token aggregation to handle NULL token columns
|
||||||
|
duration: ""
|
||||||
|
verification_result: passed
|
||||||
|
completed_at: 2026-03-30T18:23:26.734Z
|
||||||
|
blocker_discovered: false
|
||||||
|
---
|
||||||
|
|
||||||
|
# T01: Added 3 TEXT columns to pipeline_events (system_prompt_text, user_prompt_text, response_text), Redis-backed debug mode toggle, and per-stage token summary endpoint
|
||||||
|
|
||||||
|
**Added 3 TEXT columns to pipeline_events (system_prompt_text, user_prompt_text, response_text), Redis-backed debug mode toggle, and per-stage token summary endpoint**
|
||||||
|
|
||||||
|
## What Happened
|
||||||
|
|
||||||
|
Added three nullable TEXT columns to PipelineEvent for full LLM I/O capture. Created Alembic migration 006. Added debug_mode config setting. Built three new endpoints: debug-mode GET/PUT (Redis-backed with config fallback, following review_mode pattern) and token-summary aggregation. Extended event listing to include new fields. All work deployed and verified on ub01.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
Migration 006 applied successfully. Debug mode GET returns false (default), PUT toggles to true, GET reads back true. Token summary returns real per-stage aggregation for existing video data. DB columns confirmed via psql. Event listing response includes new nullable text fields.
|
||||||
|
|
||||||
|
## Verification Evidence
|
||||||
|
|
||||||
|
| # | Command | Exit Code | Verdict | Duration |
|
||||||
|
|---|---------|-----------|---------|----------|
|
||||||
|
| 1 | `ssh ub01 'docker exec chrysopedia-api alembic upgrade head'` | 0 | ✅ pass | 3000ms |
|
||||||
|
| 2 | `curl -sf http://ub01:8096/api/v1/admin/pipeline/debug-mode` | 0 | ✅ pass | 500ms |
|
||||||
|
| 3 | `curl -sf -X PUT debug-mode with {debug_mode:true}` | 0 | ✅ pass | 500ms |
|
||||||
|
| 4 | `curl -sf http://ub01:8096/api/v1/admin/pipeline/token-summary/{video_id}` | 0 | ✅ pass | 500ms |
|
||||||
|
| 5 | `psql \d pipeline_events | grep system_prompt` | 0 | ✅ pass | 500ms |
|
||||||
|
|
||||||
|
|
||||||
|
## Deviations
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
## Files Created/Modified
|
||||||
|
|
||||||
|
- `backend/models.py`
|
||||||
|
- `backend/config.py`
|
||||||
|
- `backend/schemas.py`
|
||||||
|
- `backend/routers/pipeline.py`
|
||||||
|
- `alembic/versions/006_debug_columns.py`
|
||||||
|
|
||||||
|
|
||||||
|
## Deviations
|
||||||
|
None.
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
None.
|
||||||
91
.gsd/milestones/M007/slices/S01/tasks/T02-PLAN.md
Normal file
91
.gsd/milestones/M007/slices/S01/tasks/T02-PLAN.md
Normal file
|
|
@ -0,0 +1,91 @@
|
||||||
|
---
|
||||||
|
estimated_steps: 56
|
||||||
|
estimated_files: 1
|
||||||
|
skills_used: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# T02: Wire debug-mode-aware LLM I/O capture into pipeline callback and verify end-to-end
|
||||||
|
|
||||||
|
Modify the pipeline's `_make_llm_callback` to conditionally capture full prompt and response text when debug mode is enabled. Thread system_prompt and user_prompt through to the callback at all 4 stage call sites. Rebuild worker container and verify end-to-end on ub01.
|
||||||
|
|
||||||
|
All work is on ub01 via SSH at /vmPool/r/repos/xpltdco/chrysopedia.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Add `system_prompt_text`, `user_prompt_text`, `response_text` kwargs to `_emit_event()` in `backend/pipeline/stages.py`. Pass them through to the `PipelineEvent` constructor.
|
||||||
|
|
||||||
|
2. Add a sync Redis helper at the top of `backend/pipeline/stages.py` to read debug mode:
|
||||||
|
```python
|
||||||
|
def _is_debug_mode() -> bool:
|
||||||
|
"""Check if debug mode is enabled via Redis. Falls back to config."""
|
||||||
|
try:
|
||||||
|
import redis
|
||||||
|
settings = get_settings()
|
||||||
|
r = redis.from_url(settings.redis_url)
|
||||||
|
val = r.get("chrysopedia:debug_mode")
|
||||||
|
r.close()
|
||||||
|
if val is not None:
|
||||||
|
return val.decode().lower() == "true"
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return getattr(get_settings(), 'debug_mode', False)
|
||||||
|
```
|
||||||
|
Note: uses sync `redis.from_url()` because Celery worker is sync (D004). The `redis` package is already installed (Celery broker dependency).
|
||||||
|
|
||||||
|
3. Modify `_make_llm_callback(video_id, stage)` to accept `system_prompt: str | None = None` and `user_prompt: str | None = None` parameters. Inside the callback closure:
|
||||||
|
- Check `_is_debug_mode()`
|
||||||
|
- If debug on: pass `system_prompt_text=system_prompt`, `user_prompt_text=user_prompt`, `response_text=content` to `_emit_event()`
|
||||||
|
- If debug off: pass None for all three (existing behavior preserved)
|
||||||
|
|
||||||
|
4. Update all 4 `llm.complete()` call sites to pass prompt text to the callback factory:
|
||||||
|
- Stage 2 (~line 278): `_make_llm_callback(video_id, "stage2_segmentation", system_prompt=system_prompt, user_prompt=user_prompt)`
|
||||||
|
- Stage 3 (~line 370): same pattern
|
||||||
|
- Stage 4 (~line 485): same pattern
|
||||||
|
- Stage 5 (~line 724): same pattern
|
||||||
|
|
||||||
|
5. Rebuild and restart worker + API containers:
|
||||||
|
```bash
|
||||||
|
cd /vmPool/r/repos/xpltdco/chrysopedia
|
||||||
|
docker compose build chrysopedia-api chrysopedia-worker
|
||||||
|
docker compose up -d
|
||||||
|
docker compose restart chrysopedia-web-8096 # nginx DNS cache
|
||||||
|
```
|
||||||
|
|
||||||
|
6. End-to-end verification:
|
||||||
|
a. Enable debug mode: `curl -X PUT http://ub01:8096/admin/pipeline/debug-mode -H 'Content-Type: application/json' -d '{"debug_mode":true}'`
|
||||||
|
b. Trigger pipeline for a video: `curl -X POST http://ub01:8096/admin/pipeline/trigger/{video_id}`
|
||||||
|
c. Wait for pipeline completion (check logs or events)
|
||||||
|
d. Query events: `curl http://ub01:8096/admin/pipeline/events/{video_id}?event_type=llm_call` — verify `system_prompt_text`, `user_prompt_text`, `response_text` are populated
|
||||||
|
e. Query token summary: `curl http://ub01:8096/admin/pipeline/token-summary/{video_id}` — verify per-stage breakdown
|
||||||
|
f. Disable debug mode: `curl -X PUT http://ub01:8096/admin/pipeline/debug-mode -H 'Content-Type: application/json' -d '{"debug_mode":false}'`
|
||||||
|
g. (Optional) Trigger another run — verify new events have null prompt/response columns
|
||||||
|
|
||||||
|
## Must-Haves
|
||||||
|
|
||||||
|
- [ ] `_emit_event` accepts and stores prompt/response text columns
|
||||||
|
- [ ] `_is_debug_mode()` reads from sync Redis, falls back to config
|
||||||
|
- [ ] `_make_llm_callback` conditionally captures full I/O when debug is on
|
||||||
|
- [ ] All 4 stage call sites pass system_prompt and user_prompt to callback
|
||||||
|
- [ ] Pipeline still works when debug is off (existing behavior preserved)
|
||||||
|
- [ ] End-to-end: debug on → pipeline → events have full prompt/response text
|
||||||
|
|
||||||
|
## Observability Impact
|
||||||
|
|
||||||
|
- Signals added: Full LLM system prompt, user prompt, and response text stored per llm_call event when debug mode is on
|
||||||
|
- How a future agent inspects this: `curl http://ub01:8096/admin/pipeline/events/{video_id}?event_type=llm_call` — check system_prompt_text/user_prompt_text/response_text fields
|
||||||
|
- Failure state exposed: `_is_debug_mode()` failure is silent (falls back to off) — matches existing best-effort pattern in `_emit_event`
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
- ``backend/pipeline/stages.py` — existing _emit_event, _make_llm_callback, and 4 stage call sites`
|
||||||
|
- ``backend/models.py` — PipelineEvent model with new columns (from T01)`
|
||||||
|
- ``backend/routers/pipeline.py` — debug-mode and token-summary endpoints (from T01)`
|
||||||
|
- ``backend/config.py` — Settings.debug_mode field (from T01)`
|
||||||
|
|
||||||
|
## Expected Output
|
||||||
|
|
||||||
|
- ``backend/pipeline/stages.py` — _emit_event with prompt/response params, _is_debug_mode() helper, _make_llm_callback with conditional capture, all 4 call sites updated`
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
ssh ub01 'curl -sf http://ub01:8096/admin/pipeline/debug-mode | python3 -c "import sys,json; assert json.load(sys.stdin)[\"debug_mode\"] in [True,False]"' && ssh ub01 'docker logs chrysopedia-worker --tail 5 2>&1 | head -5'
|
||||||
6
.gsd/milestones/M007/slices/S02/S02-PLAN.md
Normal file
6
.gsd/milestones/M007/slices/S02/S02-PLAN.md
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
# S02: Debug Payload Viewer — Inline View, Copy, and Export in Admin UI
|
||||||
|
|
||||||
|
**Goal:** Frontend components for viewing, copying, and exporting debug payloads. Per-stage token summary cards on the video detail view.
|
||||||
|
**Demo:** After this: Admin clicks an LLM call event, sees full prompt and response in a readable viewer, copies to clipboard, or downloads as JSON
|
||||||
|
|
||||||
|
## Tasks
|
||||||
6
.gsd/milestones/M007/slices/S03/S03-PLAN.md
Normal file
6
.gsd/milestones/M007/slices/S03/S03-PLAN.md
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
# S03: Transcript Folder Watcher — Auto-Ingest Service
|
||||||
|
|
||||||
|
**Goal:** New Python service that watches a configured directory for new .json files, validates them against the Chrysopedia transcript schema, and POSTs them to the ingest endpoint. Runs as a Docker Compose service.
|
||||||
|
**Demo:** After this: Drop a transcript JSON into the watch folder on ub01 → file is detected, validated, POSTed to /ingest, pipeline triggers automatically
|
||||||
|
|
||||||
|
## Tasks
|
||||||
6
.gsd/milestones/M007/slices/S04/S04-PLAN.md
Normal file
6
.gsd/milestones/M007/slices/S04/S04-PLAN.md
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
# S04: Admin UX Audit — Prune, Streamline, and Polish
|
||||||
|
|
||||||
|
**Goal:** Audit all admin pages (Pipeline, Reports, Review). Remove unhelpful elements. Improve pipeline page layout for the primary workflow: check status, investigate issues, retrigger. Add clear visual hierarchy for the content management workflow.
|
||||||
|
**Demo:** After this: Admin pipeline page is cleaner and more efficient for daily content management. Dead UI elements removed. Workflow improvements visible.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
6
.gsd/milestones/M007/slices/S05/S05-PLAN.md
Normal file
6
.gsd/milestones/M007/slices/S05/S05-PLAN.md
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
# S05: Key Moment Card Text Overflow Fix
|
||||||
|
|
||||||
|
**Goal:** Fix text overflow on key moment cards. Source filenames and timestamp ranges must respect card width. Apply overflow: hidden, text-overflow: ellipsis, white-space: nowrap for single-line fields. Use word-break: break-word where wrapping is appropriate.
|
||||||
|
**Demo:** After this: Key moment cards with long filenames and timestamp ranges display cleanly — text truncated with ellipsis or wrapped within card bounds, no horizontal bleed
|
||||||
|
|
||||||
|
## Tasks
|
||||||
6
.gsd/milestones/M007/slices/S06/S06-PLAN.md
Normal file
6
.gsd/milestones/M007/slices/S06/S06-PLAN.md
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
# S06: Mobile Viewport Overflow Fix — Technique Pages and Global Content
|
||||||
|
|
||||||
|
**Goal:** Fix mobile viewport overflow across content pages. Ensure all containers respect viewport width with max-width: 100%, overflow-wrap: break-word, box-sizing: border-box. Focus on technique pages (tag rows, body text, metadata) but audit all pages for the same class of issue.
|
||||||
|
**Demo:** After this: Technique page on Galaxy S25 Ultra renders cleanly — body text wraps, tags don't overflow, metadata stays within viewport, no horizontal scroll
|
||||||
|
|
||||||
|
## Tasks
|
||||||
27
CLAUDE.md
27
CLAUDE.md
|
|
@ -46,3 +46,30 @@ docker logs -f chrysopedia-worker
|
||||||
# View API logs
|
# View API logs
|
||||||
docker logs -f chrysopedia-api
|
docker logs -f chrysopedia-api
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Remote Host: hal0022 (Whisper Transcription)
|
||||||
|
|
||||||
|
- **Host alias:** `hal0022`
|
||||||
|
- **IP:** 10.0.0.131
|
||||||
|
- **OS:** Windows (domain-joined to a.xpltd.co)
|
||||||
|
- **SSH user:** `a\jlightner`
|
||||||
|
- **SSH key:** `~/.ssh/hal0022_ed25519`
|
||||||
|
- **Role:** GPU workstation for Whisper transcription of video content
|
||||||
|
|
||||||
|
### Connecting
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh hal0022
|
||||||
|
```
|
||||||
|
|
||||||
|
SSH config is already set up in `~/.ssh/config` on dev01.
|
||||||
|
|
||||||
|
### Content Location on hal0022
|
||||||
|
|
||||||
|
Video source files reside at:
|
||||||
|
|
||||||
|
```
|
||||||
|
A:\Education\Artist Streams & Content
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: This is a Windows path. When accessing via SSH, use the appropriate path format for the shell available on hal0022.
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue