# S02 Research: Stage 5 Quality Scorer & Voice Preservation Dial

## Summary

This slice adds a `score` subcommand to the existing `pipeline.quality` CLI that evaluates Stage 5 synthesis output (technique pages) across 5 dimensions, plus a `voice_level` parameter that modifies the stage 5 prompt to dial voice preservation up or down — with the scorer proving the dial works by producing measurably different voice preservation scores.

The work is **targeted research** — known patterns (LLM-as-judge scoring, prompt interpolation), applied to well-understood codebase structures.

## Recommendation

Build a `scorer.py` module in `backend/pipeline/quality/` alongside the existing `fitness.py`. The scorer:

1. Takes a reference technique page (either from DB by slug, or from a JSON file) and scores it across 5 dimensions using LLM-as-judge evaluation.
2. Optionally re-synthesizes the page with a modified prompt (voice_level dial injected into the stage 5 system prompt) before scoring.
3. Outputs a structured report with per-dimension scores and a composite.

The voice_level dial modifies the stage 5 synthesis prompt by interpolating a voice-emphasis instruction. At 0.0 ("clinical"), the prompt suppresses direct quotes and creator voice. At 1.0 ("maximum voice"), it amplifies quote preservation, creator opinions, and personality. The existing prompt (stage5_synthesis.txt) already has strong voice preservation language — that's roughly voice_level 0.6-0.7 baseline.

## Implementation Landscape

### Existing code to build on

| File | What it provides | How S02 uses it |
|------|-----------------|-----------------|
| `backend/pipeline/quality/__main__.py` | argparse CLI with subcommand pattern | Add `score` subcommand with `--slug`, `--file`, `--voice-level` args |
| `backend/pipeline/quality/fitness.py` | `TestResult` dataclass, `FitnessRunner` pattern | Follow same pattern: `ScoreRunner` class, `ScoreResult` dataclass |
| `backend/pipeline/stages.py` → `_synthesize_chunk()` | Calls LLM with stage5 prompt + moments | Scorer can reuse `_load_prompt()`, `_get_llm_client()`, `_get_stage_config()` |
| `backend/pipeline/schemas.py` → `SynthesizedPage`, `SynthesisResult` | Pydantic models for stage 5 output | Scorer validates re-synthesized output against these |
| `backend/config.py` → `Settings` | All LLM endpoint config | Scorer uses same LLM client config |
| `prompts/stage5_synthesis.txt` | The full synthesis prompt (~4KB) | Base prompt for voice_level interpolation |
| `backend/pipeline/llm_client.py` → `LLMClient.complete()` | LLM call with fallback, modality support | Used for both re-synthesis and scoring judge calls |

### The 5 scoring dimensions

Based on the stage 5 prompt's quality guidelines, these are the natural scoring dimensions:

1. **Structural quality** — Does the page have well-named sections (not generic "Overview"/"Tips"), appropriate section count, 2-5 paragraphs per section, proper body_sections structure?
2. **Content specificity** — Does the prose contain concrete details (frequencies, ratios, ms values, plugin settings) vs vague generalities? Ratio of specific claims to filler sentences.
3. **Voice preservation** — Does the page preserve the creator's actual words, opinions, warnings? Presence of direct quotes, attributed opinions, personality.
4. **Readability / flow** — Does the page read as synthesis (not concatenation)? Logical section ordering, merged related info, no redundancy, contradiction handling.
5. **Factual fidelity** — Does the page avoid inventing information not in the source moments? No phantom specifics, no hallucinated plugin names or settings.

Each dimension scored 0.0–1.0 by the LLM judge. Composite = weighted average (weights configurable, default equal).

### Voice level dial implementation

The dial works by interpolating a voice-emphasis modifier into the stage 5 system prompt before the LLM call. This is NOT a new prompt file — it's a runtime prefix/suffix injected into the existing prompt.

**voice_level 0.0 ("clinical"):** Append instruction to suppress direct quotes, write in neutral third-person encyclopedia style, avoid attributing opinions.

**voice_level 0.5 ("balanced"):** No modification — use the base prompt as-is. The existing prompt already asks for moderate voice preservation.

**voice_level 1.0 ("maximum voice"):** Append instruction to maximize direct quotes from transcript, preserve every memorable phrase, prioritize creator's exact words over paraphrase, include personality and strong opinions even if tangential.

Linear interpolation between these anchor points selects which modifier text to inject and how strongly. In practice, 3 bands (low/mid/high) with blended instructions is simpler and more reliable than continuous interpolation with an LLM.

### Scoring approach: LLM-as-judge

The scorer sends the technique page JSON + the source moments to the LLM with a scoring rubric prompt. The rubric prompt is a new file (`prompts/scoring_rubric.txt` or hardcoded in scorer.py — hardcoded is simpler for iteration, can extract later).

The judge LLM call returns a JSON object with per-dimension scores and brief justifications. Using the same LLM that generated the content is acceptable for relative scoring (comparing voice_level 0.2 vs 0.8 on the same content) even though it's not ideal for absolute quality assessment.

### Reference article sourcing

The scorer needs a technique page to evaluate. Two modes:
- `--slug <slug>`: Load from DB (requires DB connection). Loads the TechniquePage + its KeyMoments + classification data.
- `--file <path>`: Load from a JSON file (no DB needed). The file contains the synthesized page JSON + the source moments array. This mode is useful for testing without a running stack.

For the voice_level comparison demo, the scorer needs to re-synthesize from source moments, not just score an existing page. So the flow is:
1. Load reference moments (from DB or file)
2. Synthesize at voice_level X using modified prompt
3. Score the result
4. Repeat at voice_level Y
5. Compare scores

### Data needed for reference test

The scorer needs at least one set of real key moments to re-synthesize. Options:
- Connect to the production DB on ub01 and pull a technique page + its moments
- Create a fixture JSON file with sample moments for offline testing

A fixture file is better for the CLI demo — no DB dependency. The fixture should contain a realistic set of 5-8 moments with actual transcript excerpts, plugin mentions, and classification data.

### File structure after S02

```
backend/pipeline/quality/
  __init__.py          (existing)
  __main__.py          (modified — add 'score' subcommand)
  fitness.py           (existing, untouched)
  scorer.py            (NEW — ScoreRunner, ScoreResult, scoring logic)
  voice_dial.py        (NEW — voice level prompt modifier)
  fixtures/            (NEW — sample moments JSON for offline testing)
    sample_moments.json
```

### Natural task seams

1. **T01: ScoreResult dataclass + scoring rubric + ScoreRunner skeleton** — Define the 5-dimension scoring schema, write the LLM judge rubric prompt, implement `ScoreRunner.score_page()` that sends a page to the judge and returns structured scores. Wire into CLI as `score` subcommand with `--slug` and `--file` args.

2. **T02: Voice dial implementation + re-synthesis flow** — Implement `voice_dial.py` with prompt modifier logic (3 bands). Add `--voice-level` arg to the `score` subcommand. When voice_level is provided, re-synthesize the page from source moments using the modified prompt before scoring. Build the sample fixture file.

3. **T03: Integration verification + report formatting** — End-to-end test: score a reference article, run at voice_level 0.2 vs 0.8, verify scores differ meaningfully on the voice preservation dimension. Format the output report (similar to fitness report style). Verify CLI help text.

### Risks

| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| LLM judge scores don't differentiate voice levels | Medium | The scoring rubric must be specific about what constitutes voice preservation (direct quotes present, attributed opinions, personality markers). If the judge is too coarse, sharpen the rubric with examples. |
| No live LLM endpoint accessible from dev machine | High | Same as S01 — we can build and verify structure/imports locally, but actual scoring requires FYN-LLM. The fixture file allows verifying the data flow without a live endpoint. |
| Re-synthesis is expensive (full stage 5 LLM call per voice level) | Low | Expected — this is a quality tool, not a hot path. Each run takes ~30-60s per synthesis. Acceptable for a CLI benchmarking tool. |

### Skills check

No external skills needed. This is pure Python (dataclasses, Pydantic, argparse) using the existing LLM client. No new libraries required.
