---
id: S01
parent: M013
milestone: M013
provides:
  - pipeline.quality package with FitnessRunner and argparse CLI entry point
  - TestResult dataclass for structured test results
requires:
  []
affects:
  - S02
  - S03
key_files:
  - backend/pipeline/quality/__init__.py
  - backend/pipeline/quality/__main__.py
  - backend/pipeline/quality/fitness.py
key_decisions:
  - Connectivity probe as first action in run_all() for fast failure before running tests
  - Mandelbrot test uses thinking modality to exercise both LLM modes
  - Generous validation thresholds to avoid flaky failures from LLM variance
patterns_established:
  - pipeline.quality package structure with argparse subcommands — S02/S03 add score and optimize subcommands to the same CLI
  - TestResult dataclass as standard return type for all quality tests — carries name, passed, elapsed_seconds, token_count, detail
observability_surfaces:
  - none
drill_down_paths:
  - .gsd/milestones/M013/slices/S01/tasks/T01-SUMMARY.md
duration: ""
verification_result: passed
completed_at: 2026-04-01T08:46:13.884Z
blocker_discovered: false
---

# S01: General FYN-LLM Fitness Suite

**Built pipeline.quality package with FitnessRunner CLI — 9 tests across 4 categories (Mandelbrot reasoning, JSON compliance, instruction following, diverse battery), clean connectivity error handling, exits 0/1.**

## What Happened

Created the `backend/pipeline/quality/` package as the foundation for the M013 quality assurance toolkit. The package has three files: `__init__.py`, `__main__.py` (argparse CLI with `fitness` subcommand), and `fitness.py` (FitnessRunner class).

FitnessRunner implements 9 tests across 4 categories:
1. **Mandelbrot reasoning** (1 test) — asks about the Mandelbrot set area, checks for key concepts. Uses thinking modality to exercise both LLM modes.
2. **JSON compliance** (2 tests) — simple and nested JSON with Pydantic validation. Catches malformed/empty responses gracefully.
3. **Instruction following** (3 tests) — bullet count, keyword inclusion, lowercase constraint. Programmatic compliance checks.
4. **Diverse battery** (3 tests) — summarization, classification, extraction. Tests practical pipeline-relevant capabilities.

The CLI runs `python -m pipeline.quality fitness`, prints a formatted pass/fail report with per-test timing and token counts, and exits 0 (all pass) or 1 (any failure or connectivity error). A connectivity pre-check probes the LLM endpoint before running tests, providing a clear error message with the endpoint URL on failure — no tracebacks.

The argparse structure is designed for extension: S02 and S03 will add `score` and `optimize` subcommands to the same CLI.

## Verification

1. Import check: `cd backend && python -c 'from pipeline.quality.fitness import FitnessRunner; print("import ok")'` — exits 0, prints "import ok".
2. CLI connectivity error: `cd backend && python -m pipeline.quality fitness` — prints clear "Cannot reach LLM endpoint" message with URLs, exits 1, no traceback.
3. Help output: `python -m pipeline.quality --help` — shows fitness subcommand with description.
4. No-subcommand: `python -m pipeline.quality` — prints usage, exits 1.

## Requirements Advanced

None.

## Requirements Validated

None.

## New Requirements Surfaced

None.

## Requirements Invalidated or Re-scoped

None.

## Deviations

None.

## Known Limitations

Cannot verify actual test pass/fail behavior without a live FYN-LLM endpoint — only connectivity error path is testable on this machine.

## Follow-ups

None.

## Files Created/Modified

- `backend/pipeline/quality/__init__.py` — Empty package init
- `backend/pipeline/quality/__main__.py` — Argparse CLI with fitness subcommand, extensible for score/optimize
- `backend/pipeline/quality/fitness.py` — FitnessRunner class with 9 tests, 4 categories, connectivity pre-check, formatted report output
