# S04 Research: Expand to Pipeline Stages 2-4

## Depth: Targeted

Known patterns (optimization loop, variant generation, scoring) applied to new stages. Main complexity is defining stage-appropriate scoring rubrics and fixture formats.

## Summary

Extending the optimizer from stage-5-only to stages 2-4 requires four changes per stage: (1) a scoring rubric tailored to what that stage produces, (2) fixture data matching the stage's input shape, (3) format markers for variant validation, and (4) a score-variant pathway that calls the right LLM prompt and parses the right schema.

The existing architecture in `optimizer.py` and `variant_generator.py` is well-factored for this — the main hardcoding is in the scorer's `SCORING_RUBRIC` (stage-5-specific dimensions) and the optimizer's `_score_variant` method (calls synthesis → scores technique page).

## Requirement Coverage

**R013** (Prompt Template System) — already validated but this slice extends it with automated optimization for stages 2-4, strengthening the "re-run extraction on specific videos for calibration" aspect.

## Implementation Landscape

### What exists

| File | Role | Stage-5 Coupling |
|---|---|---|
| `backend/pipeline/quality/scorer.py` | `SCORING_RUBRIC`, `DIMENSIONS`, `ScoreResult`, `ScoreRunner` | SCORING_RUBRIC and DIMENSIONS are hardcoded to 5 stage-5 dimensions. ScoreResult dataclass has those 5 as named fields. `score_page()` expects page JSON shape. `synthesize_and_score()` loads stage5_synthesis.txt. |
| `backend/pipeline/quality/optimizer.py` | `OptimizationLoop`, `OptimizationResult` | `run()` loads `stage{N}_synthesis.txt` (already parameterized). `_score_variant()` calls synthesis and expects `SynthesisResult` schema. Fixture loader expects `{creator_name, moments}`. |
| `backend/pipeline/quality/variant_generator.py` | `PromptVariantGenerator`, `VARIANT_META_PROMPT` | `VARIANT_META_PROMPT` references "synthesis prompt" and synthesis-specific language. `_FORMAT_MARKERS` are `["SynthesisResult", '"pages"', "body_sections", "title", "summary"]` — stage-5-specific. |
| `backend/pipeline/quality/__main__.py` | CLI | `_run_optimize()` rejects `args.stage != 5` with an error message. |
| `backend/pipeline/quality/fixtures/sample_moments.json` | Test fixture | Stage 5 format: `{creator_name, topic_category, moments: [...]}` |

### Prompt files and schemas per stage

| Stage | Prompt File | Input Shape | Output Schema | What to Score |
|---|---|---|---|---|
| 2 (segmentation) | `stage2_segmentation.txt` | Transcript segments: `[idx] (start-end) text` | `SegmentationResult{segments: [{start_index, end_index, topic_label, summary}]}` | Coverage completeness (no gaps/overlaps), topic label specificity, segment boundary accuracy, summary quality |
| 3 (extraction) | `stage3_extraction.txt` | Topic group segments: `(start-end) text` with topic label | `ExtractionResult{moments: [{title, summary, start_time, end_time, content_type, plugins, raw_transcript}]}` | Moment richness (detail density), timestamp accuracy, content_type correctness, summary actionability, plugin name normalization |
| 4 (classification) | `stage4_classification.txt` | Moments list + taxonomy text | `ClassificationResult{classifications: [{moment_index, topic_category, topic_tags, content_type_override}]}` | Category accuracy, tag completeness, tag specificity, coverage (all moments classified), no-overlap (one category per moment) |

### Format markers per stage

Stage 2: `"segments"`, `start_index`, `end_index`, `topic_label`
Stage 3: `"moments"`, `content_type`, `raw_transcript`, `plugins`
Stage 4: `"classifications"`, `moment_index`, `topic_category`, `topic_tags`

### Architecture approach

Two viable approaches:

**A. Stage-specific scorer classes** — Create `Stage2Scorer`, `Stage3Scorer`, `Stage4Scorer` alongside the existing `ScoreRunner` (which becomes `Stage5Scorer`). Each has its own rubric, dimensions, and `score_output()` method. `OptimizationLoop` dispatches to the right scorer based on `self.stage`.

**B. Parameterized scorer with rubric registry** — Keep one `ScoreRunner` class but make it accept a rubric config (dimensions list, rubric text, output parser). A `STAGE_CONFIGS` dict maps stage number → config.

**Recommendation: B (registry).** The scoring flow is identical across stages (send rubric + output + input to LLM judge, parse scores). Only the rubric text, dimensions, format markers, and fixture→LLM-input formatting differ. A registry keeps the surface area small.

### Key design decisions

1. **ScoreResult generalization** — Currently has 5 named fields (`structural`, `content_specificity`, etc.). For stages 2-4 with different dimensions, either: (a) use a generic `scores: dict[str, float]` field, or (b) keep the named fields for stage 5 and add a generic dict for others. Option (a) is cleaner — stage 5's named fields can be populated from the dict.

2. **Fixture format per stage** — Each stage needs different fixture data:
   - Stage 2: `{transcript_segments: [{index, start_time, end_time, text}]}`
   - Stage 3: `{topic_label, segments: [{start_time, end_time, text}]}`
   - Stage 4: `{moments: [...], taxonomy: "..."}`
   
   The optimizer's `_load_fixture()` and `_score_variant()` need to be stage-aware.

3. **Variant generator meta-prompt** — The `VARIANT_META_PROMPT` currently references synthesis-specific language. It needs to be generalized or have per-stage variants. The core pattern (analyze scores → improve weakest dimensions → preserve format) is the same.

4. **Fixture creation** — Need sample fixtures for stages 2-4. Can extract from the existing sample_moments.json data (moments → stage 3/4 input) and from a real transcript (stage 2 input). Alternatively, create synthetic minimal fixtures.

### Files to modify

- `backend/pipeline/quality/scorer.py` — Add stage configs registry (rubric, dimensions, format markers per stage), generalize `ScoreResult`, add `score_stage_output()` method
- `backend/pipeline/quality/optimizer.py` — Generalize `_score_variant()` to dispatch per stage, generalize fixture loading
- `backend/pipeline/quality/variant_generator.py` — Generalize meta-prompt and format markers per stage
- `backend/pipeline/quality/__main__.py` — Remove stage-5 gate, add stage validation (2-5 only)
- `backend/pipeline/quality/fixtures/` — Add sample fixtures for stages 2-4

### Natural task seams

1. **Scorer generalization + rubric registry** — Define stage 2-4 scoring dimensions and rubrics, generalize ScoreResult, add stage config registry. This is the foundation.
2. **Optimizer + variant generator generalization** — Make OptimizationLoop stage-aware (fixture loading, score dispatch), generalize variant generator format markers and meta-prompt.
3. **Fixtures + CLI integration** — Create stage 2-4 fixtures, remove the stage-5 gate in CLI, wire everything together, verify end-to-end.

### Verification

- `python -m pipeline.quality optimize --stage 2 --iterations 1 --file <stage2_fixture>` runs without error
- `python -m pipeline.quality optimize --stage 3 --iterations 1 --file <stage3_fixture>` runs without error
- `python -m pipeline.quality optimize --stage 4 --iterations 1 --file <stage4_fixture>` runs without error
- `python -m pipeline.quality optimize --stage 5 --iterations 1 --file <stage5_fixture>` still works (no regression)
- `python -m pipeline.quality optimize --stage 6` → error message
- Import checks for all modified modules pass
