# S03 Research: Prompt Variant Generator & Automated A/B Loop

## Summary

This slice builds an automated optimization loop that generates prompt variants for Stage 5 synthesis, scores each against reference articles using the existing `ScoreRunner`, and outputs a leaderboard with score trajectories. The core deliverable is `python -m pipeline.quality optimize --stage 5 --iterations 10`.

The existing infrastructure is solid: `ScoreRunner` handles scoring, `VoiceDial` demonstrates prompt modification patterns, `LLMClient` handles LLM calls with primary/fallback. The new work is: (1) an LLM-powered prompt variant generator, (2) an optimization loop orchestrator, (3) leaderboard + trajectory reporting.

## Recommendation

Build in three tasks:

1. **Prompt Variant Generator** — `PromptVariantGenerator` class that uses the LLM to mutate a base prompt, producing N variants per iteration. This is the riskiest piece (LLM must produce valid, meaningfully different prompts).
2. **Optimization Loop** — `OptimizationLoop` class that orchestrates: load base prompt → generate variants → score each → select winners → repeat. Stores results in a simple JSON log file.
3. **CLI + Reporting** — Wire the `optimize` subcommand into `__main__.py`, add leaderboard table and score trajectory ASCII chart output.

## Implementation Landscape

### Existing Code to Build On

| File | What it provides | How S03 uses it |
|------|-----------------|-----------------|
| `backend/pipeline/quality/scorer.py` | `ScoreRunner.score_page()`, `ScoreRunner.synthesize_and_score()`, `ScoreResult` dataclass | Core scoring for each variant. `synthesize_and_score()` already chains synthesis → scoring. |
| `backend/pipeline/quality/voice_dial.py` | `VoiceDial` class with band-based prompt modification | Pattern reference for prompt modification. S03's variant generator is more flexible — LLM-driven mutations vs fixed bands. |
| `backend/pipeline/quality/__main__.py` | CLI with `fitness` and `score` subcommands | Add `optimize` subcommand here. |
| `backend/pipeline/quality/__init__.py` | `sys.path` bootstrap for project-root execution | Already handles CWD resolution. |
| `backend/pipeline/stages.py` | `_load_prompt()`, `_get_stage_config()` | Load the base prompt template, get model/modality config. Already used by `synthesize_and_score()`. |
| `backend/pipeline/schemas.py` | `SynthesisResult`, `SynthesizedPage` | Parse synthesis output. Already used by `synthesize_and_score()`. |
| `backend/pipeline/llm_client.py` | `LLMClient.complete()`, `LLMClient.parse_response()` | Drive both variant generation and synthesis calls. |
| `backend/pipeline/quality/fixtures/sample_moments.json` | 6-moment fixture file | Default test data for optimization runs. |
| `prompts/stage5_synthesis.txt` | 129-line Stage 5 synthesis prompt | Base prompt to optimize. Path: `./prompts/` from backend CWD or project root via symlink. |

### Prompt Variant Generation Strategy

The generator should use the LLM itself to produce prompt variants. Approach:

1. **Meta-prompt**: A system prompt instructing the LLM to act as a prompt engineer. Given the current best prompt and its scores, produce a variant that targets the weakest dimension(s).
2. **Mutation types**: The meta-prompt should guide the LLM toward specific mutation strategies:
   - **Targeted improvement**: Focus on the lowest-scoring dimension (e.g., if `structural` is 0.4, modify section naming instructions)
   - **Balanced tweak**: Small modifications across multiple dimensions
   - **Creative restructure**: Reorganize prompt sections or add new constraints
3. **Constraint**: Variants must preserve the core JSON output format instructions (the synthesis must still produce valid `SynthesisResult` JSON). The meta-prompt must instruct the LLM not to modify the output format section.
4. **Output**: The variant generator returns the full modified prompt text as a string.

Key risk: The LLM might produce variants that are too similar to the base or that break the output format. Mitigation: validate that the variant differs by at least N characters from the base, and that it still contains the JSON format instructions.

### Optimization Loop Design

```
OptimizationLoop:
  __init__(client, stage, fixture_path, iterations, variants_per_iter)
  
  run() → OptimizationResult:
    base_prompt = _load_prompt(f"stage{stage}_synthesis.txt")
    baseline_score = synthesize_and_score(base_prompt, moments)
    
    best = (base_prompt, baseline_score)
    history = [baseline_score]
    
    for i in range(iterations):
      variants = generator.generate(best.prompt, best.scores, n=variants_per_iter)
      for variant in variants:
        score = synthesize_and_score(variant, moments)
        history.append(score)
        if score.composite > best.composite:
          best = (variant, score)
    
    return OptimizationResult(best, history, leaderboard)
```

Design decisions to note:
- **Variants per iteration**: Default 2-3 (each requires a synthesis + scoring LLM call, so cost is ~2 calls per variant per iteration).
- **Selection strategy**: Simple greedy — keep the best-scoring variant as the seed for next iteration. No population/crossover complexity.
- **Persistence**: Write a JSON log file after each iteration with prompt text, scores, and metadata. This allows resuming and provides the data for trajectory charts.
- **Stage parameter**: Initially only `--stage 5` works. The `--stage` flag is present for S04 expansion.

### Reporting

- **Leaderboard**: Top N variants by composite score, showing per-dimension breakdown. Use formatted terminal output (existing `print_report` pattern or `rich` library which is installed).
- **Score trajectory**: ASCII line chart showing composite score across iterations. Can use a simple text-based chart (20-30 lines of code) — no need for matplotlib. The iteration count is small (10-20) so ASCII is sufficient.
- **Output file**: Save full results to a JSON file at `backend/pipeline/quality/results/` for later analysis.

### CLI Interface

```
python -m pipeline.quality optimize \
  --stage 5 \
  --iterations 10 \
  --variants-per-iter 2 \
  --file backend/pipeline/quality/fixtures/sample_moments.json \
  [--output-dir backend/pipeline/quality/results/]
```

Add to `__main__.py` alongside existing `fitness` and `score` subcommands.

### Dependencies

No new pip packages needed. `rich` is already installed for table output if desired, but plain formatted print (matching existing `print_report` style) is simpler and consistent.

### File Plan

| File | Action | Description |
|------|--------|-------------|
| `backend/pipeline/quality/variant_generator.py` | Create | `PromptVariantGenerator` class with LLM-powered prompt mutation |
| `backend/pipeline/quality/optimizer.py` | Create | `OptimizationLoop` class orchestrating generate→score→select cycle |
| `backend/pipeline/quality/__main__.py` | Modify | Add `optimize` subcommand |
| `backend/pipeline/quality/results/` | Create dir | Output directory for optimization run results |

### Verification Strategy

1. `python -m pipeline.quality optimize --help` — shows --stage, --iterations, --variants-per-iter, --file args
2. Import check: `python -c "from pipeline.quality.variant_generator import PromptVariantGenerator; print('ok')"`
3. Import check: `python -c "from pipeline.quality.optimizer import OptimizationLoop; print('ok')"`
4. Dry-run with fixture data (will fail at LLM call but should show clean error, not traceback): `python -m pipeline.quality optimize --stage 5 --iterations 1 --file backend/pipeline/quality/fixtures/sample_moments.json`
5. Unit-level: `PromptVariantGenerator` meta-prompt construction can be tested without LLM by verifying prompt assembly

### Risks

1. **LLM variant quality**: The meta-prompt for generating variants is the trickiest part. If the LLM produces near-identical variants or breaks output format, the loop produces no improvement. Mitigation: validate variant diff and format preservation.
2. **Cost per iteration**: Each variant needs synthesis + scoring = 2 LLM calls. 10 iterations × 2 variants = 40 LLM calls total. With a slow/rate-limited endpoint, this could take 10-20 minutes. Add progress output per iteration.
3. **Prompt file path resolution**: Known issue from S02 — `_load_prompt` uses `settings.prompts_path` which defaults to `./prompts`. Works from backend/ but from project root needs the symlink. Already handled by existing symlink setup.

### Natural Task Seams

1. **T01: PromptVariantGenerator** — Self-contained class in new file. Testable via import + prompt assembly check. No dependency on optimizer.
2. **T02: OptimizationLoop + CLI** — Depends on T01. Wires generator + scorer into loop, adds CLI subcommand, adds reporting/output.

Two tasks is the natural decomposition. T01 is the riskiest (meta-prompt design), T02 is integration + reporting.
