# S01 Research — Pipeline Debug Mode: Full LLM I/O Capture and Token Accounting

## Summary

The existing `PipelineEvent` model and `_make_llm_callback` infrastructure captures token counts and a truncated 2000-char `content_preview` for each LLM call. **No prompt text is stored.** Full response text is discarded. This slice needs to:

1. Add a debug mode toggle (global, Redis-backed like `review_mode`)
2. When debug is on, capture full system prompt, user prompt, and full response text for every LLM call
3. Expose per-stage token aggregation via API

The work is straightforward — extending an established event-capture pattern with two new columns, a toggle, and an aggregation query.

## Recommendation

**Targeted approach — extend existing infrastructure, don't rebuild it.**

- Add `system_prompt_text` and `user_prompt_text` TEXT columns to `pipeline_events` (alongside existing `payload` JSONB which keeps the response content). Alternatively, store full response in a dedicated `response_text` TEXT column for symmetry and queryability, keeping `payload` for metadata only.
- Debug mode: global Redis key `chrysopedia:debug_mode` (follows `review_mode` pattern from D007). Config default `debug_mode: bool = False`. When debug is **off**, behavior is unchanged (truncated preview only). When **on**, full prompts + full response stored.
- Thread prompt text into the callback by expanding `_make_llm_callback` to accept `system_prompt` and `user_prompt` parameters. The callback closure already captures `video_id` and `stage`.
- Token aggregation: add `GET /admin/pipeline/token-summary/{video_id}` that groups `pipeline_events` by `stage` and sums token columns.

## Implementation Landscape

### Existing Code (what to modify)

| File | Role | What changes |
|------|------|-------------|
| `backend/models.py` (line ~386) | `PipelineEvent` model | Add `system_prompt_text`, `user_prompt_text`, `response_text` — all `Text, nullable=True` |
| `backend/pipeline/stages.py` (line ~52) | `_emit_event()` | Accept new `system_prompt_text`, `user_prompt_text`, `response_text` kwargs, pass to model |
| `backend/pipeline/stages.py` (line ~87) | `_make_llm_callback()` | Accept `system_prompt`, `user_prompt` params; conditionally capture full text when debug mode is on |
| `backend/pipeline/stages.py` (lines 275-280, 365-372, 478-487, 720-726) | Stage `llm.complete()` call sites | Pass `system_prompt` and `user_prompt` to `_make_llm_callback()` |
| `backend/config.py` | `Settings` | Add `debug_mode: bool = False` |
| `backend/routers/pipeline.py` | Pipeline admin endpoints | Add debug mode GET/PUT, token summary endpoint, include new fields in event response |
| `backend/schemas.py` | Pydantic schemas | Add `DebugModeResponse`, `DebugModeUpdate`, `TokenSummary` schemas |
| `alembic/versions/006_debug_columns.py` | New migration | Add 3 TEXT columns to `pipeline_events` |

### Key Design Details

**Storage sizing:** System prompts are 4-12KB. User prompts include transcript text — can be 50-200KB for long videos. Full LLM responses are similar magnitude to user prompts. At ~4 LLM calls per video (stages 2-5, though stage 3 and 5 may loop), worst case is ~1-2MB per video pipeline run. PostgreSQL TEXT columns handle this fine. No need for filesystem or blob storage at this scale.

**Stage 3 and 5 looping:** Stage 3 calls LLM once per topic group; stage 5 calls once per (creator, category) group. Each call already produces a separate `PipelineEvent` with `event_type="llm_call"`. The existing infrastructure handles multi-call stages correctly — no special grouping needed.

**Debug mode check location:** The `_make_llm_callback` factory runs in the Celery worker (sync context). It needs to read debug mode. Two options:
- Read from config `settings.debug_mode` (simpler, requires worker restart to change)
- Read from Redis via sync `redis.Redis` (matches `review_mode` runtime toggle pattern, no restart needed)

**Recommendation:** Use sync Redis read in the callback. The worker already imports `config.get_settings()` which has `redis_url`. Create a thin sync Redis helper (or just `redis.from_url()` inline) — the async `redis_client.py` helper is for FastAPI only.

**`_safe_parse_llm_response` retry calls:** When the first LLM parse fails, `_safe_parse_llm_response` makes a retry call with a nudge prompt. This retry call does NOT have an `on_complete` callback — it won't emit a debug event. This is acceptable for now (retry is a diagnostic edge case), but note it for S02 if full call tracing is needed.

**Per-stage token aggregation query:**
```sql
SELECT stage,
       COUNT(*) AS call_count,
       SUM(prompt_tokens) AS total_prompt_tokens,
       SUM(completion_tokens) AS total_completion_tokens,
       SUM(total_tokens) AS total_tokens
FROM pipeline_events
WHERE video_id = :vid AND event_type = 'llm_call'
GROUP BY stage
ORDER BY MIN(created_at)
```

### Natural Task Seams

1. **DB migration + model** — Add 3 TEXT columns to `pipeline_events`, update `PipelineEvent` model. Pure schema work, verifiable with `alembic upgrade head`.
2. **Debug mode toggle** — Config setting + Redis key + GET/PUT endpoints + schemas. Follows `review_mode` pattern exactly. Verifiable via API calls.
3. **Capture instrumentation** — Modify `_make_llm_callback` to accept and conditionally store prompts/response. Modify all 4 stage call sites to pass prompt text. This is the core work. Verifiable by triggering pipeline with debug on and querying events.
4. **Token summary endpoint** — New `GET /admin/pipeline/token-summary/{video_id}` with aggregation query. Verifiable via API call after pipeline run.

Tasks 1-2 are independent. Task 3 depends on 1 (columns must exist). Task 4 depends on nothing new (uses existing columns).

### Constraints & Risks

- **Celery worker sync context:** The debug mode Redis read must use sync `redis.Redis`, not the async client from `redis_client.py`. This is the same constraint that led to sync SQLAlchemy in the worker (D004).
- **Large TEXT columns:** No index needed on prompt/response text. These are write-once, read-rarely columns for debugging. The existing `ix_pipeline_events_video_created` index handles all query patterns.
- **Migration on production data:** The `pipeline_events` table has existing rows. Adding nullable TEXT columns is a non-blocking ALTER TABLE in PostgreSQL — no data rewrite needed.

### Verification Strategy

1. Migration: `docker exec chrysopedia-api alembic upgrade head` succeeds, columns visible in `\d pipeline_events`
2. Debug toggle: `curl` GET/PUT on debug mode endpoint returns expected values
3. Full capture: Enable debug mode, trigger pipeline for a video, query events — `system_prompt_text`, `user_prompt_text`, `response_text` populated for `llm_call` events
4. Token summary: `GET /admin/pipeline/token-summary/{video_id}` returns per-stage breakdown
5. Debug off: Disable debug mode, trigger pipeline — new events have NULL prompt/response text columns (existing truncated `content_preview` in payload still works)
