# S04 Research: Pipeline Compose-or-Create Logic

## Summary

This slice modifies stage 5 (synthesis) in `backend/pipeline/stages.py` to:
1. Detect when a technique page already exists for the same creator+category (from a prior video)
2. Use the compose prompt (`stage5_compose.txt`) instead of synthesis prompt when composing
3. Store the v2 `body_sections_format` on newly created/updated pages
4. Populate the `technique_page_videos` association table so pages track all contributing videos

The compose prompt, Pydantic schemas, and test harness are all done (S01, S02). The data model and migration are done (S03). This slice wires them together in the live pipeline.

## Recommendation

Targeted research. The pattern is clear from reading the existing code — the compose-or-create decision is a conditional branch inside the per-category loop in `stage5_synthesis()`. The main risk is getting the LLM call plumbing right (same shape as `_synthesize_chunk` but using compose prompt and different user prompt format). No new technology. No ambiguous requirements.

## Implementation Landscape

### File: `backend/pipeline/stages.py` (2102 lines)

**Current flow (stage5_synthesis, line 1127):**
1. Load video, moments, creator, classification data
2. Group moments by topic_category
3. For each category group: synthesize via `_synthesize_chunk()` (or chunk+merge for large groups)
4. Persist pages: check for existing page by slug or by prior_page_ids (from Redis snapshot), create or update
5. Link moments to pages, set `processing_status = complete`

**What changes:**
- After grouping moments, before calling LLM: query DB for existing `TechniquePage` with same `creator_id` + `topic_category`
- If found → compose path: load existing page's body_sections + moments, build compose prompt via logic similar to `build_compose_prompt()` from test_harness.py, call LLM with `stage5_compose.txt` system prompt, parse result as `SynthesisResult`
- If not found → create path: existing synthesis flow (unchanged)
- After persisting page: INSERT into `technique_page_videos` (upsert pattern due to unique constraint)
- Set `body_sections_format = 'v2'` on all newly created/updated pages (both compose and create paths)

**Key helpers to reuse:**
- `_build_moments_text()` (line 957) — formats moments for prompt, returns (text, tags)
- `_synthesize_chunk()` (line 983) — single-category synthesis LLM call
- `_safe_parse_llm_response()` (line 325) — parse + truncation detection + retry
- `_load_prompt()` (line 244) — loads from prompts/ directory
- `estimate_max_tokens()` — from `pipeline.llm_client`
- `_make_llm_callback()` (line 137) — observability callback
- `_build_request_params()` (line 189) — LLM request config
- `_capture_pipeline_metadata()` (line 884) — for version snapshots

**New import needed:** `TechniquePageVideo` from `models`

### File: `backend/pipeline/test_harness.py`

Contains `build_compose_prompt()` (line 332) which is the reference implementation for building the compose user prompt. The pipeline stage will need its own version that works with real `KeyMoment` ORM objects instead of `MockKeyMoment` test doubles. The structure is identical — XML tags with `<existing_page>`, `<existing_moments>`, `<new_moments>`, `<creator>`.

### Compose prompt inputs needed:
1. **existing_page** — JSON of the existing TechniquePage's SynthesizedPage-compatible dict (title, slug, topic_category, summary, body_sections, signal_chains, plugins, source_quality)
2. **existing_moments** — formatted text of key moments already linked to this page (from previous video(s))
3. **new_moments** — formatted text of moments from the current video, with offset indices starting at N
4. **creator** — creator name

### The compose-or-create decision point:

For each `(creator_id, topic_category)` group in the current video's moments:
```
existing_page = SELECT FROM technique_pages 
  WHERE creator_id = ? AND LOWER(topic_category) = LOWER(?)
  LIMIT 1
```
- If `existing_page` exists → compose path
- If not → create path (current synthesis flow)

The existing code already does slug-based and prior_page_ids-based matching at *persist* time (line ~1320). The compose decision needs to happen *before* the LLM call, not after. This is the key architectural change — the detection moves earlier in the flow.

### TechniquePageVideo population:

After creating or updating a `TechniquePage`, insert a row:
```python
from sqlalchemy.dialects.postgresql import insert as pg_insert

stmt = pg_insert(TechniquePageVideo).values(
    technique_page_id=page.id,
    source_video_id=video_id,
).on_conflict_do_nothing()
session.execute(stmt)
```

The `on_conflict_do_nothing` handles the unique constraint gracefully for reprocessing scenarios.

### Loading existing moments for compose prompt:

When composing, we need the existing page's linked moments:
```python
existing_moments = session.execute(
    select(KeyMoment)
    .where(KeyMoment.technique_page_id == existing_page.id)
    .order_by(KeyMoment.start_time)
).scalars().all()
```

These get formatted as indices [0]-[N-1]. The new video's moments for this category get indices [N]-[N+M-1].

### body_sections_format tracking:

- New pages created via synthesis → `body_sections_format = 'v2'` (the v5 prompt always outputs v2)
- Pages updated via compose → `body_sections_format = 'v2'` (compose prompt also outputs v2)
- Existing v1 pages NOT processed in this pipeline run → unchanged (backward compatible)

The column already has `default='v1'` and `server_default='v1'` from S03 migration.

## Seams / Task Decomposition

### T01: Compose detection + compose LLM call helper
- Add `_build_compose_user_prompt()` helper (similar to test harness's `build_compose_prompt()` but using real ORM objects)
- Add `_compose_into_existing()` helper that takes an existing page + existing moments + new moments, calls LLM with compose system prompt, returns `SynthesisResult`
- This is the riskiest piece — get it testable before wiring into the main flow

### T02: Wire compose-or-create into stage5_synthesis + TechniquePageVideo
- Modify the per-category loop: before calling `_synthesize_chunk()`, check for existing page
- If found and has moments → call `_compose_into_existing()`
- If not → existing synthesis path
- After persisting page: insert `TechniquePageVideo` row
- Set `body_sections_format = 'v2'` on all pages touched
- Add `TechniquePageVideo` to imports

### T03: Integration test / verification
- Test with mock LLM to verify compose path is triggered correctly
- Verify TechniquePageVideo rows are created
- Verify body_sections_format is set
- The roadmap demo says "Process two COPYCATT videos" — this is an end-to-end verification on ub01, not something we can unit test here

## Constraints

1. **Sync SQLAlchemy only** — stages.py uses sync sessions (Celery is sync). No async.
2. **stage5_merge.txt doesn't exist** — `_merge_pages_by_slug` references it but the file is missing. This is pre-existing; don't fix it in this slice.
3. **Redis classification data** — stage 4 stores classification in Redis with 24h TTL. If the second video is processed >24h after the first, classification data for the first video may be gone. The compose flow loads existing *moments from the DB*, not classification data — so this isn't a blocker.
4. **Existing page detection must be case-insensitive** — KNOWLEDGE.md notes "LLM-generated topic categories have inconsistent casing". Use `func.lower()` for category matching.
5. **The compose prompt** (`stage5_compose.txt`) is self-contained — no runtime import from synthesis prompt needed (per S02 decision).
6. **Existing _load_prior_pages Redis snapshot** — currently used for reprocessing the *same* video. The compose flow is for a *different* video contributing to the same page. These are separate mechanisms — compose uses DB query, reprocess uses Redis snapshot. Don't conflate them.

## Risks

1. **Compose LLM output quality** — First time the compose prompt runs against real LLM. May need prompt tuning. Mitigated by: the prompt was carefully designed in S02, and `_safe_parse_llm_response` handles parse failures with retry.
2. **Multiple pages per category** — If a creator has 2+ pages in the same category (from chunked synthesis), the compose detection query returns only one. The current code handles this with `LIMIT 1` / `.first()`. Worth logging when >1 exists but proceeding with the first match.
3. **Moment count growth** — Composing many moments into a single page may exceed context limits. The existing truncation recovery (split-in-half retry) doesn't apply to compose since we can't split the existing page. Mitigated by: compose prompts are typically shorter than synthesis (existing page is already summarized).

## Verification Strategy

1. **Unit tests** for `_build_compose_user_prompt()` — XML structure, citation offset math (same pattern as test_harness_compose.py tests)
2. **Unit test** for compose-or-create branching logic — mock the DB query result
3. **Integration check** on ub01: process a video, verify page created with format v2 and TechniquePageVideo row. Then process a second video for same creator — verify compose path triggered, page updated, second TechniquePageVideo row added.
