# S03: Prompt Variant Generator & Automated A/B Loop — UAT

**Milestone:** M013
**Written:** 2026-04-01T09:12:07.490Z

## UAT: S03 — Prompt Variant Generator & Automated A/B Loop

### Preconditions
- Working directory: project root (content-to-kb-automator)
- Python environment with backend dependencies installed
- `backend/pipeline/quality/fixtures/sample_moments.json` exists (created in S02)

### Test 1: Module Imports
**Steps:**
1. Run `python -c "from pipeline.quality.variant_generator import PromptVariantGenerator; print('ok')"`
2. Run `python -c "from pipeline.quality.optimizer import OptimizationLoop, OptimizationResult; print('ok')"`

**Expected:** Both print 'ok' and exit 0.

### Test 2: CLI Help Output
**Steps:**
1. Run `python -m pipeline.quality optimize --help`

**Expected:** Output shows all 5 arguments: --stage (default 5), --iterations (default 10), --variants-per-iter (default 2), --file (required), --output-dir (default backend/pipeline/quality/results/).

### Test 3: Stage Validation — Unsupported Stage
**Steps:**
1. Run `python -m pipeline.quality optimize --stage 3 --iterations 1 --file backend/pipeline/quality/fixtures/sample_moments.json`

**Expected:** Prints error containing "only stage 5" and exits with code 1.

### Test 4: Stage Validation — Stage 2
**Steps:**
1. Run `python -m pipeline.quality optimize --stage 2 --iterations 1 --file backend/pipeline/quality/fixtures/sample_moments.json`

**Expected:** Same rejection as Test 3 — prints error containing "only stage 5" and exits 1.

### Test 5: Missing Fixture File
**Steps:**
1. Run `python -m pipeline.quality optimize --stage 5 --iterations 1 --file /nonexistent/path.json`

**Expected:** Prints error about missing file and exits with non-zero code. No traceback.

### Test 6: Missing Required --file Arg
**Steps:**
1. Run `python -m pipeline.quality optimize --stage 5 --iterations 1`

**Expected:** argparse error about required --file argument.

### Test 7: Results Directory
**Steps:**
1. Verify `backend/pipeline/quality/results/.gitkeep` exists

**Expected:** File exists, directory is tracked in git.

### Test 8: End-to-End Optimization (requires LLM connectivity)
**Preconditions:** FYN-LLM reachable at configured endpoint.
**Steps:**
1. Run `python -m pipeline.quality optimize --stage 5 --iterations 2 --variants-per-iter 1 --file backend/pipeline/quality/fixtures/sample_moments.json --output-dir /tmp/chrysopedia_test_results/`
2. Check stdout for iteration progress lines
3. Check stdout for leaderboard table (top variants by composite score)
4. Check stdout for ASCII trajectory chart
5. Check `/tmp/chrysopedia_test_results/` for a JSON file matching `optimize_stage5_*.json`
6. Validate JSON contains keys: best_prompt, best_scores, history, config, elapsed_seconds

**Expected:** Loop runs 2 iterations, generates 1 variant per iteration, scores each, prints leaderboard and trajectory, writes JSON result file with all expected keys.

### Edge Case 9: LLM Unreachable
**Steps:**
1. Set LLM endpoint to an unreachable host (e.g., modify config temporarily)
2. Run `python -m pipeline.quality optimize --stage 5 --iterations 1 --file backend/pipeline/quality/fixtures/sample_moments.json`

**Expected:** Clean error message about connectivity failure. No Python traceback shown to user.
