# S04: Pipeline Compose-or-Create Logic

**Goal:** Stage 5 detects existing technique pages by creator+category and uses the compose prompt to merge new video content into them. All pages get body_sections_format='v2' and technique_page_videos rows tracking contributing videos.
**Demo:** After this: Process two COPYCATT videos. Second video's moments composed into existing page. technique_page_videos has both video IDs.

## Tasks
- [x] **T01: Added _build_compose_user_prompt(), _compose_into_existing(), and compose-or-create branching to stage5_synthesis with body_sections_format='v2' and TechniquePageVideo tracking** — Add two helper functions to stages.py and modify the stage5_synthesis per-category loop to detect existing pages and branch to the compose path.

## Steps

1. Add `TechniquePageVideo` to the imports from `models` at line ~27.

2. Add `_build_compose_user_prompt(existing_page, existing_moments, new_moments, creator_name)` helper function:
   - Takes an existing `TechniquePage` ORM object, a list of `KeyMoment` ORM objects (existing), a list of `(KeyMoment, dict)` tuples (new moments with classification), and creator name string
   - Serialize existing page to dict matching SynthesizedPage shape: title, slug, topic_category, summary, body_sections, signal_chains, plugins, source_quality
   - Format existing moments as `[0]-[N-1]` using `_build_moments_text()` pattern but from plain KeyMoment objects (not tuples with cls_info — existing moments don't have classification data, use empty dict)
   - Format new moments as `[N]-[N+M-1]` using `_build_moments_text()` with offset indices applied
   - Build XML-tagged user prompt: `<existing_page>`, `<existing_moments>`, `<new_moments>`, `<creator>` tags (same structure as test_harness.py's `build_compose_prompt()`)
   - Return the user prompt string

3. Add `_compose_into_existing(existing_page, existing_moments, new_moment_group, category, creator_name, system_prompt, llm, model_override, modality, hard_limit, video_id, run_id)` helper function:
   - Load compose system prompt via `_load_prompt('stage5_compose.txt', video_id=video_id)`
   - Call `_build_compose_user_prompt()` to build user prompt
   - Estimate tokens via `estimate_max_tokens()`
   - Call `llm.complete()` with compose system prompt, response_model=SynthesisResult, same callback/param pattern as `_synthesize_chunk()`
   - Parse via `_safe_parse_llm_response()` and return SynthesisResult

4. Modify the per-category loop in `stage5_synthesis()` (around line 1200):
   - **Before** the existing chunked synthesis block, add compose detection:
     ```
     existing_page = session.execute(
         select(TechniquePage).where(
             TechniquePage.creator_id == video.creator_id,
             func.lower(TechniquePage.topic_category) == func.lower(category),
         )
     ).scalars().first()
     ```
   - If `existing_page` is found, load its linked moments:
     ```
     existing_moments = session.execute(
         select(KeyMoment)
         .where(KeyMoment.technique_page_id == existing_page.id)
         .order_by(KeyMoment.start_time)
     ).scalars().all()
     ```
   - If existing_page AND existing_moments → compose path: call `_compose_into_existing()`, use result.pages as synthesized_pages
   - Log INFO: 'Stage 5: Composing into existing page \'%s\' (%d existing moments + %d new moments)'
   - If >1 page matches, log WARNING about multiple matches and proceed with first
   - If no existing_page → fall through to existing synthesis block (unchanged)
   - Wrap in `else` so existing chunked synthesis only runs when not composing

5. In the persist block (around line 1380), after the `if existing:` / `else:` branch that creates/updates the page:
   - Set `page.body_sections_format = 'v2'` on every page (both new and updated)
   - Add TechniquePageVideo INSERT:
     ```python
     from sqlalchemy.dialects.postgresql import insert as pg_insert
     stmt = pg_insert(TechniquePageVideo.__table__).values(
         technique_page_id=page.id,
         source_video_id=video.id,
     ).on_conflict_do_nothing()
     session.execute(stmt)
     ```

## Must-Haves

- [ ] `_build_compose_user_prompt()` produces XML-tagged prompt with correct offset indices
- [ ] `_compose_into_existing()` calls LLM with compose system prompt and returns SynthesisResult
- [ ] Compose-or-create decision queries DB by creator_id + LOWER(topic_category)
- [ ] Existing synthesis path unchanged when no existing page found
- [ ] body_sections_format = 'v2' set on all pages
- [ ] TechniquePageVideo row inserted for every page+video combination
- [ ] Case-insensitive category matching (func.lower)

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| LLM (compose) | _safe_parse_llm_response retries once, then raises (existing retry mechanism) | Celery task retry (max_retries=3) | SynthesisResult validation rejects, retry with feedback |
| DB (existing page query) | Exception propagates to stage-level handler, triggers retry | Same | N/A |

## Negative Tests

- No existing page for creator+category → falls through to standard synthesis (no compose)
- Existing page found but zero linked moments → should still compose (empty existing_moments list)
- Multiple pages match creator+category → uses first, logs warning
  - Estimate: 1.5h
  - Files: backend/pipeline/stages.py
  - Verify: cd /home/aux/projects/content-to-kb-automator && python -c "from pipeline.stages import _build_compose_user_prompt, _compose_into_existing; print('imports OK')" && grep -q 'body_sections_format' backend/pipeline/stages.py && grep -q 'TechniquePageVideo' backend/pipeline/stages.py && grep -q 'stage5_compose' backend/pipeline/stages.py
- [x] **T02: Added 12 unit tests covering compose prompt construction, branching logic, body_sections_format, TechniquePageVideo insertion, and case-insensitive category matching** — Create test_compose_pipeline.py covering compose prompt construction, compose-or-create branching, TechniquePageVideo insertion, and body_sections_format setting.

## Steps

1. Create `backend/pipeline/test_compose_pipeline.py`.

2. Write test fixtures:
   - Mock KeyMoment objects (using simple namedtuples or dataclasses with .title, .summary, .content_type, .start_time, .end_time, .plugins, .raw_transcript, .id, .technique_page_id, .source_video_id)
   - Mock TechniquePage object with .id, .title, .slug, .topic_category, .summary, .body_sections, .signal_chains, .plugins, .source_quality, .creator_id
   - Use `unittest.mock` for DB session and LLM client

3. Test `_build_compose_user_prompt()`:
   - **test_compose_prompt_xml_structure**: verify output contains `<existing_page>`, `<existing_moments>`, `<new_moments>`, `<creator>` tags
   - **test_compose_prompt_offset_indices**: with 3 existing moments and 2 new moments, verify existing use [0]-[2] and new use [3]-[4]
   - **test_compose_prompt_empty_existing_moments**: 0 existing, N new → new moments start at [0]
   - **test_compose_prompt_page_json**: verify existing page serialized as JSON within `<existing_page>` tags

4. Test compose-or-create branching:
   - **test_compose_branch_triggered**: mock session.execute to return an existing page + moments for the same creator+category → verify `_compose_into_existing` is called (patch it)
   - **test_create_branch_no_existing**: mock session.execute to return None for existing page query → verify `_synthesize_chunk` is called instead
   - **test_category_case_insensitive**: verify query uses func.lower for category matching (inspect the query or test with mixed-case input)

5. Test TechniquePageVideo and body_sections_format:
   - **test_body_sections_format_v2**: verify pages created by both compose and create paths have body_sections_format='v2'
   - **test_technique_page_video_inserted**: verify INSERT with on_conflict_do_nothing is executed after page persist

## Must-Haves

- [ ] At least 4 tests for _build_compose_user_prompt covering XML structure, offset math, empty existing, page JSON
- [ ] At least 2 tests for branching logic (compose triggered vs create fallback)
- [ ] At least 1 test for body_sections_format = 'v2'
- [ ] At least 1 test for TechniquePageVideo insertion
- [ ] All tests pass with `python -m pytest backend/pipeline/test_compose_pipeline.py -v`
  - Estimate: 1h
  - Files: backend/pipeline/test_compose_pipeline.py
  - Verify: cd /home/aux/projects/content-to-kb-automator && python -m pytest backend/pipeline/test_compose_pipeline.py -v
