---
id: M013
title: "Prompt Quality Toolkit — LLM Fitness, Scoring, and Automated Optimization"
status: complete
completed_at: 2026-04-01T09:29:57.707Z
key_decisions:
  - Hardcoded scoring rubric in scorer.py rather than external prompt file — faster iteration during quality toolkit development
  - Three discrete voice bands (low/mid/high) at 0.33/0.67 boundaries instead of continuous interpolation
  - OptimizationLoop bypasses VoiceDial — owns full prompt text directly to avoid double-application
  - STAGE_CONFIGS registry pattern for centralized per-stage config (rubric, dimensions, format markers, fixture keys, prompt file, schema class)
  - Backward-compat properties on ScoreResult instead of migrating all callers when generalizing from named fields to scores dict
  - Meta-prompt pattern: LLM acts as prompt engineer receiving current prompt + scores + rubric to generate variants
key_files:
  - backend/pipeline/quality/__init__.py
  - backend/pipeline/quality/__main__.py
  - backend/pipeline/quality/fitness.py
  - backend/pipeline/quality/scorer.py
  - backend/pipeline/quality/voice_dial.py
  - backend/pipeline/quality/variant_generator.py
  - backend/pipeline/quality/optimizer.py
  - backend/pipeline/quality/fixtures/sample_moments.json
  - backend/pipeline/quality/fixtures/sample_segments.json
  - backend/pipeline/quality/fixtures/sample_topic_group.json
  - backend/pipeline/quality/fixtures/sample_classifications.json
lessons_learned:
  - Project-root symlinks + sys.path bootstrap in __init__.py solve CWD-dependent import issues for CLI tools that live inside a subdirectory (backend/) but need to run from the project root
  - Meta-prompt pattern (LLM-as-prompt-engineer) works well for variant generation when the meta-prompt includes the current prompt text, per-dimension scores, and the scoring rubric summary — gives the LLM enough context to target weak dimensions
  - Variant validation gates (min-diff threshold + format marker checks) are essential to catch trivial LLM mutations that would waste scoring budget
  - STAGE_CONFIGS registry centralizes per-stage config and makes adding new stages mechanical — better than switch/case dispatch scattered across multiple files
  - Backward-compat properties on dataclasses (e.g., .structural returning scores['structural']) allow generalization without a migration of all callers
---

# M013: Prompt Quality Toolkit — LLM Fitness, Scoring, and Automated Optimization

**Built a complete prompt quality toolkit: LLM fitness testing, 5-dimension scoring with voice preservation dial, and automated A/B prompt optimization loops for all pipeline stages 2-5.**

## What Happened

M013 delivered the `pipeline.quality` package — a CLI toolkit for testing LLM fitness, scoring pipeline output quality, and running automated prompt optimization loops.

**S01** laid the foundation: `FitnessRunner` with 9 tests across 4 categories (Mandelbrot reasoning, JSON compliance, instruction following, diverse battery). The CLI structure uses argparse subcommands designed for extension. Connectivity pre-check gives clear errors before wasting LLM calls.

**S02** added the scoring engine: `ScoreRunner` with a 5-dimension LLM-as-judge rubric (structural, content_specificity, voice_preservation, readability, factual_fidelity) and `VoiceDial` for 3-band prompt modification (low/mid/high). The `score` subcommand accepts fixture files or slugs with optional `--voice-level` to test voice preservation at different intensities. A project-root symlink and sys.path bootstrap were added to support running the CLI from any CWD.

**S03** built the optimization loop: `PromptVariantGenerator` uses a meta-prompt to have the LLM act as a prompt engineer, generating variants targeting the weakest scoring dimensions. `OptimizationLoop` iterates generate→score→select cycles, capturing full history in `OptimizationResult`. The `optimize` subcommand outputs a leaderboard table, ASCII trajectory chart, and persists results as timestamped JSON.

**S04** generalized everything from stage-5-only to stages 2-5. A `STAGE_CONFIGS` registry maps each stage to per-stage scoring rubrics, dimension lists, format markers, fixture key requirements, prompt file paths, and schema classes. `ScoreResult` was generalized from named fields to a `scores: dict[str, float]` with backward-compat properties. Stage-specific fixture files were created for stages 2-4. The optimizer dispatches per-stage user prompts and schema parsing via `_build_user_prompt()`.

The entire toolkit runs via `python -m pipeline.quality {fitness|score|optimize}` with clean error handling, no tracebacks on connectivity failures, and exit codes suitable for CI integration.

## Success Criteria Results

The roadmap defines success through the vision statement and per-slice demos:

- ✅ **FYN-LLM fitness testing**: `python -m pipeline.quality fitness` runs 9 tests across 4 categories with pass/fail output (S01)
- ✅ **Multi-dimension quality scoring**: `score` subcommand scores pipeline output across 5 dimensions (structural, content_specificity, voice_preservation, readability, factual_fidelity) (S02)
- ✅ **Voice preservation dial**: `--voice-level` parameter modifies prompts via 3-band VoiceDial, producing meaningfully different voice preservation scores at different levels (S02)
- ✅ **Prompt variant generation**: LLM-powered meta-prompt generates variants targeting weakest dimensions, with validation gates for trivial mutations (S03)
- ✅ **Automated A/B optimization loop**: `optimize` subcommand runs unattended generate→score→select iterations with leaderboard and trajectory output (S03)
- ✅ **Multi-stage support**: Optimization works for stages 2-5 with per-stage rubrics, fixtures, and schema dispatch (S04)
- ✅ **Reports**: Leaderboard table, ASCII trajectory chart, and timestamped JSON persistence (S03/S04)

## Definition of Done Results

- ✅ All 4 slices complete (S01, S02, S03, S04)
- ✅ All 4 slice summaries exist
- ✅ Code changes verified: 2,402 lines across 14 files
- ✅ Cross-slice integration: S01 package structure extended by S02 (score subcommand), S03 (optimize subcommand), S04 (multi-stage generalization) — all share the same CLI entry point and import chain
- ✅ CLI runs from both project root and backend/ directory via symlink + sys.path bootstrap

## Requirement Outcomes

- **R013 (Prompt Template System)**: Already validated. M013 extends R013 by adding automated prompt optimization — generates variants of the editable prompt templates and scores them. No status change needed since R013 was already validated; M013 advances it further by providing tooling to improve prompt quality systematically.

## Deviations

S02 added an unplanned project-root symlink and sys.path bootstrap to support running the CLI from the project root, not just from backend/. S03's OptimizationLoop does its own synthesis instead of delegating to ScoreRunner.synthesize_and_score() to avoid double-application of VoiceDial. S04 fixed a T01 fixture_keys mismatch (moments vs key_moments) in a subsequent task.

## Follow-ups

Consider moving the hardcoded scoring rubric from scorer.py to an external config file once the rubric stabilizes. The --slug path for loading test data from the database is stubbed but not implemented. QdrantManager deterministic UUID issue (from KNOWLEDGE.md) should be addressed before running optimization results against production data.
