# S01: General FYN-LLM Fitness Suite

**Goal:** A CLI command `python -m pipeline.quality fitness` that tests FYN-LLM fitness across four categories (Mandelbrot reasoning, JSON compliance, instruction following, diverse prompt battery), prints a pass/fail report with timing, and exits 0/1.
**Demo:** After this: Run `python -m pipeline.quality fitness` — outputs pass/fail for Mandelbrot question, JSON compliance, instruction following, and diverse prompt battery against live FYN-LLM

## Tasks
- [x] **T01: Built pipeline.quality package with FitnessRunner (9 tests, 4 categories) and argparse CLI that prints formatted pass/fail report and exits 0/1** — Create `backend/pipeline/quality/` package with `__init__.py`, `__main__.py` (argparse CLI with `fitness` subcommand), and `fitness.py` (FitnessRunner class implementing all four test categories). Reuse LLMClient from `pipeline/llm_client.py` and Settings from `config.py`.

## Steps

1. Create `backend/pipeline/quality/__init__.py` (empty)
2. Create `backend/pipeline/quality/__main__.py` with argparse dispatcher — `fitness` subcommand calls `FitnessRunner.run_all()`. Future subcommands (score, optimize) will be added in S02/S03.
3. Create `backend/pipeline/quality/fitness.py` with `FitnessRunner` class:
   - Constructor takes `LLMClient` instance
   - `run_all()` method runs all tests, prints formatted report, returns exit code
   - `test_mandelbrot()` — asks about the Mandelbrot set area, checks response contains key concepts (area approximately 1.5065, not exactly known). Use generous keyword matching.
   - `test_json_compliance_simple()` — requests JSON with a simple Pydantic model (e.g., {"name": str, "count": int}), verifies valid JSON parse and model validation
   - `test_json_compliance_nested()` — requests JSON with nested model, same validation
   - `test_instruction_following()` — 2-3 tests: respond in exactly N bullet points, include a specific word, use only lowercase. Check compliance programmatically.
   - `test_summarization()` — give a paragraph, ask for 2-sentence summary, verify response length is reasonable
   - `test_classification()` — give text, ask to classify from fixed list, verify response is one of the options
   - `test_extraction()` — give structured text, ask to extract specific fields as JSON, verify fields present
   - Each test method returns a `TestResult` dataclass with name, passed, elapsed_seconds, token_count, detail (failure reason)
4. Wire the entry point: `__main__.py` instantiates `LLMClient(get_settings())`, creates `FitnessRunner`, calls `run_all()`, exits with returned code
5. Handle connectivity errors: wrap the initial LLM call in a try/except for `openai.APIConnectionError` and `openai.APITimeoutError` — print a clear "Cannot reach LLM endpoint at {url}" message and exit 1
6. Test both modalities: include at least one test using `modality='thinking'` (for the thinking model) since the pipeline uses both chat and thinking modes
7. Verify: `cd backend && python -c 'from pipeline.quality.fitness import FitnessRunner'` succeeds (import check). Run `cd backend && python -m pipeline.quality fitness 2>&1; echo "exit: $?"` — should print connectivity error and exit 1 (no live LLM on this machine) rather than a traceback.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| LLM endpoint | Catch `APIConnectionError`, print clear message, exit 1 | Catch `APITimeoutError`, same handling | Log raw response, mark test as FAIL with detail |

## Negative Tests

- **Unreachable endpoint**: Running with default localhost config (no LLM running) should print a connectivity error, not a traceback
- **Malformed LLM response**: If the LLM returns non-JSON for a JSON test, the test should FAIL with a clear message, not crash
- **Empty response**: If the LLM returns empty string, tests should FAIL gracefully
  - Estimate: 1h
  - Files: backend/pipeline/quality/__init__.py, backend/pipeline/quality/__main__.py, backend/pipeline/quality/fitness.py
  - Verify: cd backend && python -c 'from pipeline.quality.fitness import FitnessRunner; print("import ok")' && python -m pipeline.quality fitness 2>&1; echo "exit: $?"
