# S01: Pipeline Debug Mode — Full LLM I/O Capture and Token Accounting

**Goal:** Debug mode toggle (Redis-backed) controls full LLM I/O capture in the pipeline. When on, every LLM call stores system prompt, user prompt, and full response text. Per-stage token summary available via API.
**Demo:** After this: Toggle debug mode on for a video, trigger pipeline, see full prompt + response stored for every LLM call with per-stage token breakdown

## Tasks
- [x] **T01: Added 3 TEXT columns to pipeline_events (system_prompt_text, user_prompt_text, response_text), Redis-backed debug mode toggle, and per-stage token summary endpoint** — Set up all infrastructure for debug mode: Alembic migration adding 3 TEXT columns to pipeline_events, PipelineEvent model update, debug mode GET/PUT endpoints (Redis-backed, following review_mode pattern from D007), token summary aggregation endpoint, and extended event listing response.

All work is on ub01 via SSH at /vmPool/r/repos/xpltdco/chrysopedia.

## Steps

1. Add `system_prompt_text`, `user_prompt_text`, `response_text` columns (all `Text, nullable=True`) to `PipelineEvent` model in `backend/models.py` after the existing `payload` column.

2. Create Alembic migration `alembic/versions/006_debug_columns.py` that adds the 3 TEXT columns to `pipeline_events`. Use `op.add_column()` with `nullable=True` — no data rewrite needed.

3. Add `debug_mode: bool = False` to `Settings` class in `backend/config.py`.

4. Add Pydantic schemas in `backend/schemas.py`:
   - `DebugModeResponse(BaseModel)` with `debug_mode: bool`
   - `DebugModeUpdate(BaseModel)` with `debug_mode: bool`
   - `TokenStageSummary(BaseModel)` with `stage: str`, `call_count: int`, `total_prompt_tokens: int`, `total_completion_tokens: int`, `total_tokens: int`
   - `TokenSummaryResponse(BaseModel)` with `video_id: str`, `stages: list[TokenStageSummary]`, `grand_total_tokens: int`

5. Add 3 new endpoints in `backend/routers/pipeline.py`:
   - `GET /admin/pipeline/debug-mode` — read from Redis key `chrysopedia:debug_mode`, fall back to `settings.debug_mode`. Follow exact pattern from review router's `get_mode()`.
   - `PUT /admin/pipeline/debug-mode` — write to Redis. Follow exact pattern from review router's `set_mode()`.
   - `GET /admin/pipeline/token-summary/{video_id}` — query `pipeline_events` grouped by stage where `event_type='llm_call'`, sum token columns, return `TokenSummaryResponse`.

6. Extend the existing `list_pipeline_events` response dict to include `system_prompt_text`, `user_prompt_text`, `response_text` fields (nullable).

7. Run migration: `docker exec chrysopedia-api alembic upgrade head`. Rebuild API container if needed: `docker compose build chrysopedia-api && docker compose up -d chrysopedia-api`.

8. Verify: curl debug-mode GET returns `{"debug_mode": false}`, PUT sets to true, GET reads back true. Curl token-summary for any video_id returns valid (possibly empty) response.

## Must-Haves

- [ ] 3 TEXT columns added to pipeline_events table via migration
- [ ] PipelineEvent model has matching mapped columns
- [ ] Debug mode GET/PUT endpoints work with Redis (fall back to config)
- [ ] Token summary endpoint returns per-stage aggregation
- [ ] Event listing includes new fields

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| Redis (debug mode read) | Fall back to config.debug_mode default | Same as error — use default | N/A (simple GET/SET of string) |
| PostgreSQL (token summary) | Return 500 with error detail | Standard DB timeout handling | N/A (aggregation query) |
  - Estimate: 1.5h
  - Files: backend/models.py, backend/config.py, backend/schemas.py, backend/routers/pipeline.py, alembic/versions/006_debug_columns.py
  - Verify: ssh ub01 'docker exec chrysopedia-api alembic upgrade head' && curl -sf http://ub01:8096/admin/pipeline/debug-mode | grep debug_mode
- [x] **T02: Wired debug-mode-aware LLM I/O capture into pipeline callback — _emit_event accepts prompt/response text, _is_debug_mode reads sync Redis, _make_llm_callback conditionally captures full I/O, all 4 stage call sites updated** — Modify the pipeline's `_make_llm_callback` to conditionally capture full prompt and response text when debug mode is enabled. Thread system_prompt and user_prompt through to the callback at all 4 stage call sites. Rebuild worker container and verify end-to-end on ub01.

All work is on ub01 via SSH at /vmPool/r/repos/xpltdco/chrysopedia.

## Steps

1. Add `system_prompt_text`, `user_prompt_text`, `response_text` kwargs to `_emit_event()` in `backend/pipeline/stages.py`. Pass them through to the `PipelineEvent` constructor.

2. Add a sync Redis helper at the top of `backend/pipeline/stages.py` to read debug mode:
   ```python
   def _is_debug_mode() -> bool:
       """Check if debug mode is enabled via Redis. Falls back to config."""
       try:
           import redis
           settings = get_settings()
           r = redis.from_url(settings.redis_url)
           val = r.get("chrysopedia:debug_mode")
           r.close()
           if val is not None:
               return val.decode().lower() == "true"
       except Exception:
           pass
       return getattr(get_settings(), 'debug_mode', False)
   ```
   Note: uses sync `redis.from_url()` because Celery worker is sync (D004). The `redis` package is already installed (Celery broker dependency).

3. Modify `_make_llm_callback(video_id, stage)` to accept `system_prompt: str | None = None` and `user_prompt: str | None = None` parameters. Inside the callback closure:
   - Check `_is_debug_mode()`
   - If debug on: pass `system_prompt_text=system_prompt`, `user_prompt_text=user_prompt`, `response_text=content` to `_emit_event()`
   - If debug off: pass None for all three (existing behavior preserved)

4. Update all 4 `llm.complete()` call sites to pass prompt text to the callback factory:
   - Stage 2 (~line 278): `_make_llm_callback(video_id, "stage2_segmentation", system_prompt=system_prompt, user_prompt=user_prompt)`
   - Stage 3 (~line 370): same pattern
   - Stage 4 (~line 485): same pattern  
   - Stage 5 (~line 724): same pattern

5. Rebuild and restart worker + API containers:
   ```bash
   cd /vmPool/r/repos/xpltdco/chrysopedia
   docker compose build chrysopedia-api chrysopedia-worker
   docker compose up -d
   docker compose restart chrysopedia-web-8096  # nginx DNS cache
   ```

6. End-to-end verification:
   a. Enable debug mode: `curl -X PUT http://ub01:8096/admin/pipeline/debug-mode -H 'Content-Type: application/json' -d '{"debug_mode":true}'`
   b. Trigger pipeline for a video: `curl -X POST http://ub01:8096/admin/pipeline/trigger/{video_id}`
   c. Wait for pipeline completion (check logs or events)
   d. Query events: `curl http://ub01:8096/admin/pipeline/events/{video_id}?event_type=llm_call` — verify `system_prompt_text`, `user_prompt_text`, `response_text` are populated
   e. Query token summary: `curl http://ub01:8096/admin/pipeline/token-summary/{video_id}` — verify per-stage breakdown
   f. Disable debug mode: `curl -X PUT http://ub01:8096/admin/pipeline/debug-mode -H 'Content-Type: application/json' -d '{"debug_mode":false}'`
   g. (Optional) Trigger another run — verify new events have null prompt/response columns

## Must-Haves

- [ ] `_emit_event` accepts and stores prompt/response text columns
- [ ] `_is_debug_mode()` reads from sync Redis, falls back to config
- [ ] `_make_llm_callback` conditionally captures full I/O when debug is on
- [ ] All 4 stage call sites pass system_prompt and user_prompt to callback
- [ ] Pipeline still works when debug is off (existing behavior preserved)
- [ ] End-to-end: debug on → pipeline → events have full prompt/response text

## Observability Impact

- Signals added: Full LLM system prompt, user prompt, and response text stored per llm_call event when debug mode is on
- How a future agent inspects this: `curl http://ub01:8096/admin/pipeline/events/{video_id}?event_type=llm_call` — check system_prompt_text/user_prompt_text/response_text fields
- Failure state exposed: `_is_debug_mode()` failure is silent (falls back to off) — matches existing best-effort pattern in `_emit_event`
  - Estimate: 1.5h
  - Files: backend/pipeline/stages.py
  - Verify: ssh ub01 'curl -sf http://ub01:8096/admin/pipeline/debug-mode | python3 -c "import sys,json; assert json.load(sys.stdin)[\"debug_mode\"] in [True,False]"' && ssh ub01 'docker logs chrysopedia-worker --tail 5 2>&1 | head -5'
