---
id: S01
parent: M007
milestone: M007
provides:
  - Debug mode toggle API (GET/PUT /api/v1/admin/pipeline/debug-mode)
  - Full LLM I/O stored in pipeline_events when debug on
  - Per-stage token summary API (GET /api/v1/admin/pipeline/token-summary/{video_id})
  - Extended event listing with system_prompt_text, user_prompt_text, response_text fields
requires:
  []
affects:
  - S02
key_files:
  - backend/models.py
  - backend/config.py
  - backend/schemas.py
  - backend/routers/pipeline.py
  - alembic/versions/006_debug_columns.py
  - backend/pipeline/stages.py
key_decisions:
  - Followed review_mode Redis pattern (D007) for debug_mode toggle — consistent approach across all admin toggles
  - Debug mode checked once at callback creation time to avoid Redis round-trip per LLM call
  - Full response_text stored without truncation when debug on; existing content_preview in JSONB payload kept for non-debug use
  - Used SQL coalesce for NULL token columns in aggregation query
patterns_established:
  - Redis-backed admin toggle with config fallback: GET reads Redis key, falls back to settings default; PUT writes to Redis. Pattern now used by review_mode and debug_mode.
  - Conditional pipeline instrumentation: _make_llm_callback checks mode once at creation, closure captures the decision, no per-call overhead when off.
observability_surfaces:
  - GET /api/v1/admin/pipeline/debug-mode — current debug toggle state
  - GET /api/v1/admin/pipeline/token-summary/{video_id} — per-stage token accounting
  - pipeline_events.system_prompt_text/user_prompt_text/response_text — full LLM I/O when debug mode was on during pipeline run
drill_down_paths:
  - .gsd/milestones/M007/slices/S01/tasks/T01-SUMMARY.md
  - .gsd/milestones/M007/slices/S01/tasks/T02-SUMMARY.md
duration: ""
verification_result: passed
completed_at: 2026-03-30T18:57:12.919Z
blocker_discovered: false
---

# S01: Pipeline Debug Mode — Full LLM I/O Capture and Token Accounting

**Added debug mode toggle (Redis-backed) that captures full LLM system prompt, user prompt, and response text in pipeline_events, plus per-stage token summary endpoint.**

## What Happened

Two tasks delivered the full debug mode feature.

T01 added infrastructure: Alembic migration 006 with 3 nullable TEXT columns (system_prompt_text, user_prompt_text, response_text) on pipeline_events, PipelineEvent model update, debug_mode config setting, Redis-backed GET/PUT toggle endpoints (following the review_mode pattern from D007), and a token-summary aggregation endpoint that groups by stage.

T02 wired the capture into the pipeline: _emit_event extended to accept the 3 text fields, _is_debug_mode() reads sync Redis with config fallback, _make_llm_callback conditionally captures full I/O when debug is on (checked once at callback creation, not per-call), and all 4 stage call sites (stages 2-5) pass system_prompt and user_prompt through. The existing content_preview in the JSONB payload is preserved for non-debug use.

End-to-end verified on ub01: debug mode toggled on, pipeline ran for a video, events have populated system_prompt_text (7752 chars), user_prompt_text (971 chars), and response_text. Token summary returns per-stage breakdown (e.g., stage2: 1 call / 16K tokens, stage3: 15 calls / 66K tokens, grand total 83K). When debug is off, fields remain NULL — no storage overhead.

## Verification

All slice-level checks passed:

1. **Migration**: `docker exec chrysopedia-api alembic upgrade head` — 006_debug_columns applied, 3 TEXT columns confirmed via psql \d pipeline_events
2. **Debug mode toggle**: GET returns {"debug_mode":false}, PUT sets to true, GET reads back true, PUT sets to false, GET confirms false — full round-trip
3. **Token summary**: GET /api/v1/admin/pipeline/token-summary/{video_id} returns per-stage aggregation with call_count, prompt/completion/total tokens, and grand_total_tokens
4. **Event listing**: GET /api/v1/admin/pipeline/events/{video_id} items include system_prompt_text, user_prompt_text, response_text (populated for llm_call events when debug was on, NULL otherwise)
5. **End-to-end capture**: Video 7afc5a42 processed with debug on — stage3_extraction events contain 7752 chars of system prompt and 971 chars of user prompt
6. **Worker health**: chrysopedia-worker running without import errors after rebuild

## Requirements Advanced

None.

## Requirements Validated

None.

## New Requirements Surfaced

None.

## Requirements Invalidated or Re-scoped

None.

## Deviations

T02 noted endpoint paths need /api/v1/ prefix (omitted in plan) and Docker compose web service name is chrysopedia-web not chrysopedia-web-8096. Minor naming discrepancies, no functional impact.

## Known Limitations

Debug mode is global (all videos), not per-video. Full prompt/response text stored without truncation — for very long prompts this could grow the pipeline_events table significantly if debug is left on permanently. Intended for diagnostic use, not always-on.

## Follow-ups

S02 builds the admin UI viewer for these debug payloads — inline view, copy, and export.

## Files Created/Modified

- `backend/models.py` — Added system_prompt_text, user_prompt_text, response_text columns to PipelineEvent
- `backend/config.py` — Added debug_mode: bool = False to Settings
- `backend/schemas.py` — Added DebugModeResponse, DebugModeUpdate, TokenStageSummary, TokenSummaryResponse schemas
- `backend/routers/pipeline.py` — Added debug-mode GET/PUT endpoints, token-summary endpoint, extended event listing response
- `alembic/versions/006_debug_columns.py` — Migration adding 3 TEXT columns to pipeline_events
- `backend/pipeline/stages.py` — Added _is_debug_mode(), extended _emit_event and _make_llm_callback for conditional I/O capture, updated 4 stage call sites
