---
id: S03
parent: M013
milestone: M013
provides:
  - optimize CLI subcommand for automated prompt A/B testing
  - PromptVariantGenerator for LLM-powered prompt mutation
  - OptimizationLoop + OptimizationResult for iterative optimization with full history
  - Leaderboard table, ASCII trajectory chart, and timestamped JSON result output
requires:
  - slice: S02
    provides: ScoreRunner, ScoreResult, LLMClient, 5-dimension scoring rubric, VoiceDial pattern
affects:
  - S04
key_files:
  - backend/pipeline/quality/variant_generator.py
  - backend/pipeline/quality/optimizer.py
  - backend/pipeline/quality/__main__.py
  - backend/pipeline/quality/results/.gitkeep
key_decisions:
  - OptimizationLoop bypasses VoiceDial — owns the full prompt text directly to avoid double-application
  - Variant validation uses both length diff and line-level symmetric difference to catch trivial mutations
  - Reporting functions live in __main__.py rather than a separate reporting.py — keeps surface area small
patterns_established:
  - Meta-prompt pattern: LLM acts as prompt engineer, receives current prompt + per-dimension scores + rubric summary, outputs a modified variant targeting weakest dimensions
  - Variant validation gate: min-diff threshold + format marker check before scoring, invalid variants logged and skipped
  - Optimization history capture: full iteration×variant matrix stored in OptimizationResult for downstream leaderboard/charting/JSON export
observability_surfaces:
  - none
drill_down_paths:
  - .gsd/milestones/M013/slices/S03/tasks/T01-SUMMARY.md
  - .gsd/milestones/M013/slices/S03/tasks/T02-SUMMARY.md
duration: ""
verification_result: passed
completed_at: 2026-04-01T09:12:07.490Z
blocker_discovered: false
---

# S03: Prompt Variant Generator & Automated A/B Loop

**Automated prompt optimization loop: LLM-powered variant generation, iterative scoring, CLI with leaderboard/trajectory output, and JSON result persistence.**

## What Happened

Built two core modules and wired them into the existing quality CLI. `variant_generator.py` provides `PromptVariantGenerator` — given a base prompt and its per-dimension scores, it calls the LLM with a meta-prompt to generate N variants targeting the weakest scoring dimensions. Each variant is validated: must differ from the base by ≥50 chars (via line-level symmetric difference), must preserve JSON format markers (`SynthesisResult`, `"pages"`). Invalid variants are logged and skipped.

`optimizer.py` provides `OptimizationLoop` — loads the base stage 5 prompt and fixture data, runs a baseline score, then iterates: generate variants → score each via synthesis + the existing 5-dimension scorer → keep the best → repeat. The loop handles LLM errors gracefully (errored variants are skipped, not fatal). `OptimizationResult` captures the full history (iteration, variant index, prompt text, all scores) for downstream reporting.

The `optimize` CLI subcommand accepts `--stage`, `--iterations`, `--variants-per-iter`, `--file`, and `--output-dir`. Stage validation restricts to stage 5 (others print a clear message and exit 1). After the loop completes, three reporting functions fire: `print_leaderboard()` shows top 5 variants ranked by composite score with per-dimension breakdown; `print_trajectory()` renders a 15-row ASCII chart of best composite per iteration; `write_results_json()` persists the full result with config metadata to a timestamped JSON file.

One deviation from the plan: `OptimizationLoop._score_variant()` does its own synthesis call rather than delegating to `ScoreRunner.synthesize_and_score()`, because the loop owns the full prompt text directly and bypassing VoiceDial avoids double-application of voice modifiers.

## Verification

All slice-level verification checks passed:

1. `python -c "from pipeline.quality.variant_generator import PromptVariantGenerator; print('generator ok')"` → exit 0
2. `python -c "from pipeline.quality.optimizer import OptimizationLoop, OptimizationResult; print('optimizer ok')"` → exit 0
3. `python -m pipeline.quality optimize --help` → shows all 5 args (--stage, --iterations, --variants-per-iter, --file, --output-dir)
4. `python -m pipeline.quality optimize --stage 3 --iterations 1 --file ...` → prints "only stage 5 is supported" and exits 1
5. `backend/pipeline/quality/results/.gitkeep` exists

## Requirements Advanced

- R013 — Prompt optimization loop provides automated mechanism to improve prompt templates — generates variants, scores them, and identifies winners

## Requirements Validated

None.

## New Requirements Surfaced

None.

## Requirements Invalidated or Re-scoped

None.

## Deviations

OptimizationLoop._score_variant() performs its own synthesis call instead of delegating to ScoreRunner.synthesize_and_score(). This avoids double-application of VoiceDial modifiers since the optimization loop owns the full prompt text directly.

## Known Limitations

Only stage 5 optimization is supported — other stages print an error and exit 1. This is by design; S04 will extend to stages 2-4.

## Follow-ups

S04 extends optimization to pipeline stages 2-4 with stage-appropriate scoring dimensions.

## Files Created/Modified

- `backend/pipeline/quality/variant_generator.py` — New module: PromptVariantGenerator with meta-prompt, LLM-powered variant generation, and validation (min-diff + format markers)
- `backend/pipeline/quality/optimizer.py` — New module: OptimizationLoop (generate→score→select cycles) and OptimizationResult dataclass with full history
- `backend/pipeline/quality/__main__.py` — Added optimize subparser, print_leaderboard(), print_trajectory(), write_results_json() reporting functions
- `backend/pipeline/quality/results/.gitkeep` — Created results output directory
