# S02: Stage 5 Quality Scorer & Voice Preservation Dial

**Goal:** A `score` CLI subcommand that evaluates Stage 5 synthesis output across 5 quality dimensions using LLM-as-judge, plus a `--voice-level` parameter that modifies the synthesis prompt to dial voice preservation intensity — with the scorer proving the dial works by producing measurably different voice preservation scores at different levels.
**Demo:** After this: Run scorer on a reference article — outputs composite score across 5 dimensions. Run same article at voice_level 0.2 vs 0.8 — voice preservation score differs meaningfully

## Tasks
- [x] **T01: Built ScoreRunner with 5-dimension LLM-as-judge scoring rubric, CLI score subcommand, and realistic 6-moment sample fixture** — ## Description

Create the scorer module that evaluates a Stage 5 technique page across 5 quality dimensions using an LLM judge call. Wire it into the existing CLI as the `score` subcommand. Build a realistic fixture file for offline testing.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| LLM endpoint | Print clear connectivity error with URL, exit 1 (same as fitness) | Same — openai timeout maps to APITimeoutError | Parse judge JSON response; if malformed, log raw excerpt and return zero scores with detail message |

## Steps

1. Read `backend/pipeline/quality/fitness.py` to understand the TestResult/FitnessRunner pattern and report formatting.
2. Create `backend/pipeline/quality/scorer.py` with:
   - `ScoreResult` dataclass: `structural`, `content_specificity`, `voice_preservation`, `readability`, `factual_fidelity` (each float 0.0–1.0), `composite` (weighted average, default equal weights), `justifications` (dict of dimension → string), `elapsed_seconds` (float).
   - `ScoreRunner` class taking an `LLMClient` in __init__.
   - `ScoreRunner.score_page(page_json: dict, moments: list[dict]) -> ScoreResult` method:
     - Builds a scoring rubric prompt (hardcoded string in scorer.py — not a separate prompt file) that asks the LLM to evaluate the page against the source moments across the 5 dimensions.
     - The rubric should specify what each dimension measures (see S02-RESEARCH.md for definitions) and instruct the LLM to return JSON: `{"structural": 0.8, "content_specificity": 0.7, ..., "justifications": {"structural": "...", ...}}`.
     - Calls `self.client.complete()` with modality="chat", response_model=BaseModel (JSON mode).
     - Parses JSON response, validates all 5 dimension keys present and values in [0.0, 1.0].
     - Computes composite as mean of 5 dimensions.
     - Returns `ScoreResult`.
   - `ScoreRunner.print_report(result: ScoreResult)` — formatted report matching fitness report style: header bar, per-dimension score with justification excerpt, composite score, timing.
3. Create `backend/pipeline/quality/fixtures/sample_moments.json` with:
   - `{"creator_name": "ExampleCreator", "topic_category": "Sound design", "moments": [...]}` — 5-6 realistic moments with `summary`, `transcript_excerpt`, `topic_tags`, `topic_category`, `start_time`, `end_time` fields. Content about a concrete music production technique (e.g., snare layering or bass resampling). Include direct quotes and specific plugin/setting mentions so voice preservation scoring has signal.
4. Create `backend/pipeline/quality/fixtures/__init__.py` (empty).
5. Update `backend/pipeline/quality/__main__.py`:
   - Add `score` subcommand to argparse with args: `--file` (path to moments JSON), `--slug` (technique slug — placeholder, just store the arg, actual DB loading deferred), `--voice-level` (float, optional, default None — wired in T02).
   - When `args.command == "score"`: validate that exactly one of --file or --slug is provided. If --file, load JSON, extract `moments` and `creator_name`. Create `ScoreRunner(client)`. If `--voice-level` is None, call `score_page()` directly with the moments. Print the report.
   - For now, the --slug path prints "DB loading not yet implemented" and exits 1.
6. Verify: `cd backend && python -c "from pipeline.quality.scorer import ScoreRunner, ScoreResult; print('import ok')"` exits 0.
7. Verify: `cd backend && python -m pipeline.quality score --help` shows --file, --slug, --voice-level.
8. Verify: `cd backend && python -m pipeline.quality score --file pipeline/quality/fixtures/sample_moments.json` — hits connectivity error, prints endpoint URL, exits 1 with no traceback.

## Must-Haves

- [ ] ScoreResult dataclass with 5 float dimensions + composite + justifications + elapsed_seconds
- [ ] ScoreRunner.score_page() sends rubric + page + moments to LLM, parses JSON response
- [ ] Formatted report output with per-dimension scores and justifications
- [ ] Fixture JSON with 5+ realistic moments including transcript excerpts and plugin mentions
- [ ] `score` subcommand wired in __main__.py with --file, --slug, --voice-level args
- [ ] Connectivity error handled cleanly (same pattern as fitness)

## Verification

- `cd backend && python -c "from pipeline.quality.scorer import ScoreRunner, ScoreResult; print('import ok')"` — exits 0
- `cd backend && python -m pipeline.quality score --help` — shows all three args
- `cd backend && python -m pipeline.quality score --file pipeline/quality/fixtures/sample_moments.json` — connectivity error with URL, exit 1, no traceback
- `cd backend && python -c "import json; d=json.load(open('pipeline/quality/fixtures/sample_moments.json')); assert 'moments' in d and len(d['moments']) >= 5"` — fixture valid
  - Estimate: 1.5h
  - Files: backend/pipeline/quality/scorer.py, backend/pipeline/quality/__main__.py, backend/pipeline/quality/fixtures/sample_moments.json, backend/pipeline/quality/fixtures/__init__.py
  - Verify: cd backend && python -c "from pipeline.quality.scorer import ScoreRunner, ScoreResult; print('import ok')" && python -m pipeline.quality score --help && python -c "import json; d=json.load(open('pipeline/quality/fixtures/sample_moments.json')); assert 'moments' in d and len(d['moments']) >= 5"
- [x] **T02: Added VoiceDial class with 3-band prompt modification and ScoreRunner.synthesize_and_score() that re-synthesizes from source moments at a given voice_level before scoring** — ## Description

Build the voice dial module that modifies the stage 5 synthesis prompt based on a voice_level parameter (0.0–1.0), and wire it into the scorer so `--voice-level` triggers re-synthesis from source moments before scoring. This completes the slice by enabling the key demo: running the scorer at voice_level 0.2 vs 0.8 produces measurably different voice preservation scores.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| LLM endpoint (re-synthesis) | Same connectivity error pattern — print URL, exit 1 | Same | If synthesis returns unparseable JSON, log raw excerpt and exit 1 with message |
| LLM endpoint (scoring judge) | Same as T01 scorer | Same | Same as T01 — zero scores with detail |
| prompts/stage5_synthesis.txt | FileNotFoundError caught, print "Prompt file not found: {path}", exit 1 | N/A | N/A |

## Steps

1. Read the existing `prompts/stage5_synthesis.txt` to understand what voice language is already present (the research says it's roughly voice_level 0.6-0.7 baseline).
2. Read `backend/pipeline/stages.py` functions `_load_prompt()`, `_get_stage_config()`, and `_synthesize_chunk()` to understand the exact synthesis call pattern — system_prompt + `<creator>...</creator>\n<moments>...</moments>` user prompt format.
3. Create `backend/pipeline/quality/voice_dial.py` with:
   - `VoiceDial` class:
     - `__init__(self, base_prompt: str)` — stores the base stage 5 system prompt.
     - `modify(self, voice_level: float) -> str` — returns the modified system prompt.
       - 3 bands: low (0.0–0.33), mid (0.34–0.66), high (0.67–1.0).
       - Low band: append instruction to suppress direct quotes, write in neutral third-person encyclopedia style, avoid attributing opinions, minimize personality markers.
       - Mid band: return base prompt unmodified (the existing prompt already has moderate voice preservation).
       - High band: append instruction to maximize direct quotes from transcript, preserve every memorable phrase, prioritize creator's exact words over paraphrase, include personality and strong opinions.
     - Band boundaries at 0.33 and 0.67. Within each band, no continuous interpolation — just the band's modifier.
4. Add `ScoreRunner.synthesize_and_score(moments: list[dict], creator_name: str, voice_level: float) -> ScoreResult` method to `backend/pipeline/quality/scorer.py`:
   - Loads stage5_synthesis.txt via `_load_prompt('stage5_synthesis.txt')` (import from pipeline.stages).
   - Creates `VoiceDial(base_prompt)` and calls `modify(voice_level)` to get modified prompt.
   - Gets stage config via `_get_stage_config(5)` for model_override and modality.
   - Builds user prompt in the same format as `_synthesize_chunk`: `<creator>{name}</creator>\n<moments>\n{moments_json}\n</moments>`.
   - Calls `self.client.complete()` with the modified prompt, parses response as SynthesisResult using `self.client.parse_response(raw, SynthesisResult)`.
   - If synthesis returns valid pages, takes the first page and calls `self.score_page()` on it.
   - Returns the ScoreResult.
5. Update `backend/pipeline/quality/__main__.py` — in the `score` command handler:
   - If `--voice-level` is provided and `--file` is used: load moments from file, call `runner.synthesize_and_score(moments, creator_name, voice_level)` instead of `score_page()`.
   - If `--voice-level` is provided without `--file`: error — voice-level requires moments input (--file or --slug with DB).
6. Verify: `cd backend && python -c "from pipeline.quality.voice_dial import VoiceDial; print('import ok')"` exits 0.
7. Verify: `cd backend && python -c "from pipeline.quality.voice_dial import VoiceDial; vd = VoiceDial('base prompt'); low = vd.modify(0.1); mid = vd.modify(0.5); high = vd.modify(0.9); assert low != mid; assert high != mid; assert 'suppress' in low.lower() or 'neutral' in low.lower(); assert 'quote' in high.lower() or 'direct' in high.lower(); print('dial ok')"` — voice dial produces distinct prompts per band.
8. Verify: `cd backend && python -m pipeline.quality score --file pipeline/quality/fixtures/sample_moments.json --voice-level 0.3` — connectivity error, exit 1, no traceback.

## Must-Haves

- [ ] VoiceDial class with 3 bands (low/mid/high) producing distinct prompt modifications
- [ ] Low band suppresses voice, high band amplifies it, mid band passes through unmodified
- [ ] ScoreRunner.synthesize_and_score() re-synthesizes from moments using modified prompt, then scores
- [ ] --voice-level wired into CLI and triggers re-synthesis flow
- [ ] Stage 5 prompt loaded from prompts/stage5_synthesis.txt (not hardcoded)
- [ ] Synthesis output parsed as SynthesisResult (reuses existing schema)

## Verification

- `cd backend && python -c "from pipeline.quality.voice_dial import VoiceDial; print('import ok')"` — exits 0
- `cd backend && python -c "from pipeline.quality.voice_dial import VoiceDial; vd = VoiceDial('base'); assert vd.modify(0.1) != vd.modify(0.5) != vd.modify(0.9); print('bands ok')"` — three distinct outputs
- `cd backend && python -m pipeline.quality score --file pipeline/quality/fixtures/sample_moments.json --voice-level 0.3` — connectivity error, exit 1, no traceback
  - Estimate: 1.5h
  - Files: backend/pipeline/quality/voice_dial.py, backend/pipeline/quality/scorer.py, backend/pipeline/quality/__main__.py
  - Verify: cd backend && python -c "from pipeline.quality.voice_dial import VoiceDial; vd = VoiceDial('base'); assert vd.modify(0.1) != vd.modify(0.5); assert vd.modify(0.5) != vd.modify(0.9); print('bands ok')" && python -m pipeline.quality score --help
