---
estimated_steps: 41
estimated_files: 4
skills_used: []
---

# T01: Build ScoreRunner with 5-dimension LLM-as-judge scoring and CLI integration

## Description

Create the scorer module that evaluates a Stage 5 technique page across 5 quality dimensions using an LLM judge call. Wire it into the existing CLI as the `score` subcommand. Build a realistic fixture file for offline testing.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| LLM endpoint | Print clear connectivity error with URL, exit 1 (same as fitness) | Same — openai timeout maps to APITimeoutError | Parse judge JSON response; if malformed, log raw excerpt and return zero scores with detail message |

## Steps

1. Read `backend/pipeline/quality/fitness.py` to understand the TestResult/FitnessRunner pattern and report formatting.
2. Create `backend/pipeline/quality/scorer.py` with:
   - `ScoreResult` dataclass: `structural`, `content_specificity`, `voice_preservation`, `readability`, `factual_fidelity` (each float 0.0–1.0), `composite` (weighted average, default equal weights), `justifications` (dict of dimension → string), `elapsed_seconds` (float).
   - `ScoreRunner` class taking an `LLMClient` in __init__.
   - `ScoreRunner.score_page(page_json: dict, moments: list[dict]) -> ScoreResult` method:
     - Builds a scoring rubric prompt (hardcoded string in scorer.py — not a separate prompt file) that asks the LLM to evaluate the page against the source moments across the 5 dimensions.
     - The rubric should specify what each dimension measures (see S02-RESEARCH.md for definitions) and instruct the LLM to return JSON: `{"structural": 0.8, "content_specificity": 0.7, ..., "justifications": {"structural": "...", ...}}`.
     - Calls `self.client.complete()` with modality="chat", response_model=BaseModel (JSON mode).
     - Parses JSON response, validates all 5 dimension keys present and values in [0.0, 1.0].
     - Computes composite as mean of 5 dimensions.
     - Returns `ScoreResult`.
   - `ScoreRunner.print_report(result: ScoreResult)` — formatted report matching fitness report style: header bar, per-dimension score with justification excerpt, composite score, timing.
3. Create `backend/pipeline/quality/fixtures/sample_moments.json` with:
   - `{"creator_name": "ExampleCreator", "topic_category": "Sound design", "moments": [...]}` — 5-6 realistic moments with `summary`, `transcript_excerpt`, `topic_tags`, `topic_category`, `start_time`, `end_time` fields. Content about a concrete music production technique (e.g., snare layering or bass resampling). Include direct quotes and specific plugin/setting mentions so voice preservation scoring has signal.
4. Create `backend/pipeline/quality/fixtures/__init__.py` (empty).
5. Update `backend/pipeline/quality/__main__.py`:
   - Add `score` subcommand to argparse with args: `--file` (path to moments JSON), `--slug` (technique slug — placeholder, just store the arg, actual DB loading deferred), `--voice-level` (float, optional, default None — wired in T02).
   - When `args.command == "score"`: validate that exactly one of --file or --slug is provided. If --file, load JSON, extract `moments` and `creator_name`. Create `ScoreRunner(client)`. If `--voice-level` is None, call `score_page()` directly with the moments. Print the report.
   - For now, the --slug path prints "DB loading not yet implemented" and exits 1.
6. Verify: `cd backend && python -c "from pipeline.quality.scorer import ScoreRunner, ScoreResult; print('import ok')"` exits 0.
7. Verify: `cd backend && python -m pipeline.quality score --help` shows --file, --slug, --voice-level.
8. Verify: `cd backend && python -m pipeline.quality score --file pipeline/quality/fixtures/sample_moments.json` — hits connectivity error, prints endpoint URL, exits 1 with no traceback.

## Must-Haves

- [ ] ScoreResult dataclass with 5 float dimensions + composite + justifications + elapsed_seconds
- [ ] ScoreRunner.score_page() sends rubric + page + moments to LLM, parses JSON response
- [ ] Formatted report output with per-dimension scores and justifications
- [ ] Fixture JSON with 5+ realistic moments including transcript excerpts and plugin mentions
- [ ] `score` subcommand wired in __main__.py with --file, --slug, --voice-level args
- [ ] Connectivity error handled cleanly (same pattern as fitness)

## Verification

- `cd backend && python -c "from pipeline.quality.scorer import ScoreRunner, ScoreResult; print('import ok')"` — exits 0
- `cd backend && python -m pipeline.quality score --help` — shows all three args
- `cd backend && python -m pipeline.quality score --file pipeline/quality/fixtures/sample_moments.json` — connectivity error with URL, exit 1, no traceback
- `cd backend && python -c "import json; d=json.load(open('pipeline/quality/fixtures/sample_moments.json')); assert 'moments' in d and len(d['moments']) >= 5"` — fixture valid

## Inputs

- `backend/pipeline/quality/__main__.py`
- `backend/pipeline/quality/fitness.py`
- `backend/pipeline/llm_client.py`
- `backend/pipeline/schemas.py`

## Expected Output

- `backend/pipeline/quality/scorer.py`
- `backend/pipeline/quality/__main__.py`
- `backend/pipeline/quality/fixtures/sample_moments.json`
- `backend/pipeline/quality/fixtures/__init__.py`

## Verification

cd backend && python -c "from pipeline.quality.scorer import ScoreRunner, ScoreResult; print('import ok')" && python -m pipeline.quality score --help && python -c "import json; d=json.load(open('pipeline/quality/fixtures/sample_moments.json')); assert 'moments' in d and len(d['moments']) >= 5"
