# S03: Prompt Variant Generator & Automated A/B Loop

**Goal:** Automated prompt optimization loop that generates LLM-powered prompt variants for Stage 5 synthesis, scores each against reference articles, and outputs a ranked leaderboard with score trajectories.
**Demo:** After this: Run `python -m pipeline.quality optimize --stage 5 --iterations 10` — generates prompt variants, scores each against reference articles, outputs leaderboard and score trajectory chart

## Tasks
- [x] **T01: Created PromptVariantGenerator (LLM-powered prompt mutation) and OptimizationLoop (iterative generate→score→select engine) with full error tolerance and progress reporting** — Create two new modules: `variant_generator.py` with an LLM-powered prompt mutation engine, and `optimizer.py` with an optimization loop that orchestrates generate→score→select cycles.

## Context

The existing `ScoreRunner.synthesize_and_score()` already chains Stage 5 synthesis through a modified prompt and scores the result. The `VoiceDial` class demonstrates the pattern of appending modifier text to the base prompt. This task builds the higher-level automation: an LLM generates prompt variants targeting weak dimensions, and a loop selects winners across iterations.

## Key Interfaces

- `LLMClient.complete(system_prompt, user_prompt, response_model, modality)` — drives both variant generation and synthesis
- `ScoreRunner.synthesize_and_score(moments, creator_name, voice_level)` — scores a variant (uses voice_level=0.5 for neutral baseline)
- `_load_prompt('stage5_synthesis.txt')` — loads the base prompt to optimize
- `_get_stage_config(5)` — returns (model_override, modality) for Stage 5

## Steps

1. Create `backend/pipeline/quality/variant_generator.py`:
   - Define a meta-prompt constant (`VARIANT_META_PROMPT`) that instructs the LLM to act as a prompt engineer. Given the current best prompt, its per-dimension scores, and the scoring rubric summary, produce a single modified variant targeting the weakest dimension(s). The meta-prompt MUST instruct the LLM to preserve the JSON output format section of the synthesis prompt unchanged.
   - `PromptVariantGenerator.__init__(self, client: LLMClient)` — stores the LLM client
   - `generate(self, base_prompt: str, scores: ScoreResult, n: int = 2) -> list[str]` — calls the LLM `n` times with the meta-prompt, each time asking for a variant. Validates each variant: must differ from base by ≥50 chars, must still contain the JSON format instruction markers (e.g. `SynthesisResult` or `"pages"` key reference). Returns list of valid variant prompt strings. Logs and skips invalid variants.

2. Create `backend/pipeline/quality/optimizer.py`:
   - `@dataclass OptimizationResult`: best_prompt (str), best_score (ScoreResult), history (list of dicts with iteration, variant_index, prompt_text, score fields), elapsed_seconds (float)
   - `OptimizationLoop.__init__(self, client: LLMClient, stage: int, fixture_path: str, iterations: int, variants_per_iter: int)` — stores config, creates ScoreRunner and PromptVariantGenerator internally
   - `run(self) -> OptimizationResult` — loads base prompt via `_load_prompt(f'stage{stage}_synthesis.txt')`, loads fixture data from JSON file, runs baseline `synthesize_and_score()`, then iterates: generate N variants → score each → keep best → repeat. Prints progress per iteration. Stores full history. Returns OptimizationResult.
   - Handle LLM errors gracefully: if a variant's synthesis/scoring returns a ScoreResult with `error`, log it and skip that variant (don't abort the loop).

3. Verify both modules import cleanly from project root.

## Must-Haves

- Meta-prompt instructs LLM to preserve JSON output format section
- Variant validation: minimum diff from base, format markers present
- Invalid/errored variants are skipped, not fatal
- Progress output per iteration
- OptimizationResult contains full history for downstream reporting
  - Estimate: 1.5h
  - Files: backend/pipeline/quality/variant_generator.py, backend/pipeline/quality/optimizer.py
  - Verify: python -c "from pipeline.quality.variant_generator import PromptVariantGenerator; print('generator ok')" && python -c "from pipeline.quality.optimizer import OptimizationLoop, OptimizationResult; print('optimizer ok')"
- [x] **T02: Added optimize CLI subcommand with leaderboard table, ASCII trajectory chart, and timestamped results JSON output** — Add the `optimize` subcommand to `__main__.py` that connects PromptVariantGenerator + OptimizationLoop to the CLI, and add formatted reporting: a leaderboard table and an ASCII score trajectory chart. Write results to a JSON file.

## Context

T01 produced `PromptVariantGenerator` and `OptimizationLoop` with `OptimizationResult`. This task wires them into the existing CLI (which already has `fitness` and `score` subcommands) and adds human-readable output.

## Steps

1. Add `optimize` subparser to `__main__.py`:
   - `--stage` (int, default 5) — pipeline stage to optimize
   - `--iterations` (int, default 10) — number of optimization iterations
   - `--variants-per-iter` (int, default 2) — variants generated per iteration
   - `--file` (str, required) — path to moments JSON fixture
   - `--output-dir` (str, default `backend/pipeline/quality/results/`) — where to write result JSON
   - Validate --stage is 5 (others not yet supported, print message and exit 1 for other values)
   - Create output dir if it doesn't exist
   - Instantiate `OptimizationLoop` with parsed args, call `run()`, handle result

2. Add reporting functions (can be in `__main__.py` or a small `reporting.py`):
   - `print_leaderboard(result: OptimizationResult)` — formatted table showing top 5 variants by composite score, with per-dimension breakdown. Use the same visual style as `ScoreRunner.print_report()` (plain print with alignment, score bars).
   - `print_trajectory(result: OptimizationResult)` — ASCII line chart of composite score across iterations. ~20 rows height, iteration index on x-axis. Simple text rendering (no external deps).
   - Print both after the optimization loop completes.

3. Write results JSON:
   - Save `OptimizationResult` to `{output_dir}/optimize_stage{N}_{timestamp}.json` with: best_prompt, best_scores, full history, config (stage, iterations, variants_per_iter, fixture_path), elapsed_seconds.
   - Create `backend/pipeline/quality/results/.gitkeep` so the directory is tracked.

4. Verify the full CLI flow:
   - `python -m pipeline.quality optimize --help` shows all args
   - `python -m pipeline.quality optimize --stage 5 --iterations 1 --file backend/pipeline/quality/fixtures/sample_moments.json` runs and either produces output (if LLM reachable) or shows clean connectivity error
   - `python -m pipeline.quality optimize --stage 3` prints 'only stage 5 supported' and exits 1

## Must-Haves

- --stage, --iterations, --variants-per-iter, --file, --output-dir CLI args
- Stage validation (only 5 supported currently)
- Leaderboard table printed after loop
- ASCII trajectory chart printed after loop
- Results JSON written to output directory
- Clean error handling (no tracebacks on LLM connectivity failure)
  - Estimate: 1h
  - Files: backend/pipeline/quality/__main__.py, backend/pipeline/quality/results/.gitkeep
  - Verify: python -m pipeline.quality optimize --help && python -m pipeline.quality optimize --stage 3 --iterations 1 --file backend/pipeline/quality/fixtures/sample_moments.json 2>&1 | grep -q 'stage 5'
