---
estimated_steps: 27
estimated_files: 3
skills_used: []
---

# T01: Build the FYN-LLM fitness test suite CLI

Create `backend/pipeline/quality/` package with `__init__.py`, `__main__.py` (argparse CLI with `fitness` subcommand), and `fitness.py` (FitnessRunner class implementing all four test categories). Reuse LLMClient from `pipeline/llm_client.py` and Settings from `config.py`.

## Steps

1. Create `backend/pipeline/quality/__init__.py` (empty)
2. Create `backend/pipeline/quality/__main__.py` with argparse dispatcher — `fitness` subcommand calls `FitnessRunner.run_all()`. Future subcommands (score, optimize) will be added in S02/S03.
3. Create `backend/pipeline/quality/fitness.py` with `FitnessRunner` class:
   - Constructor takes `LLMClient` instance
   - `run_all()` method runs all tests, prints formatted report, returns exit code
   - `test_mandelbrot()` — asks about the Mandelbrot set area, checks response contains key concepts (area approximately 1.5065, not exactly known). Use generous keyword matching.
   - `test_json_compliance_simple()` — requests JSON with a simple Pydantic model (e.g., {"name": str, "count": int}), verifies valid JSON parse and model validation
   - `test_json_compliance_nested()` — requests JSON with nested model, same validation
   - `test_instruction_following()` — 2-3 tests: respond in exactly N bullet points, include a specific word, use only lowercase. Check compliance programmatically.
   - `test_summarization()` — give a paragraph, ask for 2-sentence summary, verify response length is reasonable
   - `test_classification()` — give text, ask to classify from fixed list, verify response is one of the options
   - `test_extraction()` — give structured text, ask to extract specific fields as JSON, verify fields present
   - Each test method returns a `TestResult` dataclass with name, passed, elapsed_seconds, token_count, detail (failure reason)
4. Wire the entry point: `__main__.py` instantiates `LLMClient(get_settings())`, creates `FitnessRunner`, calls `run_all()`, exits with returned code
5. Handle connectivity errors: wrap the initial LLM call in a try/except for `openai.APIConnectionError` and `openai.APITimeoutError` — print a clear "Cannot reach LLM endpoint at {url}" message and exit 1
6. Test both modalities: include at least one test using `modality='thinking'` (for the thinking model) since the pipeline uses both chat and thinking modes
7. Verify: `cd backend && python -c 'from pipeline.quality.fitness import FitnessRunner'` succeeds (import check). Run `cd backend && python -m pipeline.quality fitness 2>&1; echo "exit: $?"` — should print connectivity error and exit 1 (no live LLM on this machine) rather than a traceback.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| LLM endpoint | Catch `APIConnectionError`, print clear message, exit 1 | Catch `APITimeoutError`, same handling | Log raw response, mark test as FAIL with detail |

## Negative Tests

- **Unreachable endpoint**: Running with default localhost config (no LLM running) should print a connectivity error, not a traceback
- **Malformed LLM response**: If the LLM returns non-JSON for a JSON test, the test should FAIL with a clear message, not crash
- **Empty response**: If the LLM returns empty string, tests should FAIL gracefully

## Inputs

- ``backend/pipeline/llm_client.py` — LLMClient class with complete() method, LLMResponse wrapper, primary/fallback endpoint logic`
- ``backend/config.py` — Settings class with llm_api_url, llm_model, llm_api_key, get_settings() factory`

## Expected Output

- ``backend/pipeline/quality/__init__.py` — empty package init`
- ``backend/pipeline/quality/__main__.py` — argparse CLI with fitness subcommand`
- ``backend/pipeline/quality/fitness.py` — FitnessRunner class with all test methods and formatted report output`

## Verification

cd backend && python -c 'from pipeline.quality.fitness import FitnessRunner; print("import ok")' && python -m pipeline.quality fitness 2>&1; echo "exit: $?"
