# S01 Research — General FYN-LLM Fitness Suite

## Summary

This slice creates a new CLI module at `pipeline.quality` that runs a battery of tests against the live FYN-LLM endpoint and reports pass/fail for each. No active requirements are owned — this is new infrastructure for M013. The work is straightforward: make HTTP calls to an OpenAI-compatible API, validate responses against expected patterns, report results.

## Recommendation

Build a `pipeline/quality/` package with `__main__.py` (argparse CLI) and `fitness.py` (test runner). Reuse the existing `LLMClient` from `pipeline/llm_client.py` — it already handles primary/fallback endpoints and JSON mode. Use argparse (stdlib), not click, to avoid adding dependencies.

## Implementation Landscape

### Entry Point

The demo says `python -m pipeline.quality fitness`. This requires:
- `backend/pipeline/quality/__init__.py`
- `backend/pipeline/quality/__main__.py` — argparse dispatcher
- `backend/pipeline/quality/fitness.py` — test battery logic

Must be runnable from `backend/` directory (where `pipeline` is a package). The `config.py` Settings class loads from env vars / `.env`, so the fitness tool gets LLM config for free via `get_settings()`.

### LLM Client Reuse

`pipeline/llm_client.py` provides `LLMClient` with:
- `complete(system_prompt, user_prompt, response_model=None, modality="chat", model_override=None)` → `LLMResponse` (str subclass with `finish_reason`, `truncated`)
- `parse_response(text, model)` → validated Pydantic model
- Primary/fallback endpoint logic
- JSON mode via `response_format: {"type": "json_object"}`
- Think-tag stripping for reasoning models

The fitness suite should instantiate `LLMClient(get_settings())` and call `complete()` directly. No need to go through Celery or the pipeline stages.

### Model Configuration

From `config.py`:
- **Primary:** `llm_api_url` (default `http://localhost:11434/v1`), model `fyn-llm-agent-chat`
- **Fallback:** `llm_fallback_url`, model `fyn-llm-agent-chat`
- **Thinking model:** `fyn-llm-agent-think` (used by stages 3, 5)
- **Temperature:** 0.0 (deterministic)

The fitness suite should test both `chat` and `thinking` modalities since the pipeline uses both.

### Four Test Categories

1. **Mandelbrot question** — Ask "Is the area of the Mandelbrot set known?" or similar. Expects a coherent, factually correct answer. Pass/fail based on response containing key facts (area ≈ 1.5065..., exactly known = no). This tests basic reasoning capability.

2. **JSON compliance** — Send a prompt requesting structured JSON output with `response_model` set. Verify: (a) response is valid JSON, (b) parses against expected Pydantic schema. Test with both simple and nested schemas.

3. **Instruction following** — Give specific formatting constraints: "respond in exactly 3 bullet points", "use only lowercase", "include the word 'chrysopedia' in your response". Check compliance programmatically.

4. **Diverse prompt battery** — Multiple prompts exercising different capabilities relevant to the pipeline:
   - Summarization (give text, ask for summary)
   - Classification (give text, ask for category from a fixed list)
   - Extraction (give text, ask for specific fields)
   - Long-context handling (send a large prompt, verify coherent response)

### Output Format

CLI should print a clear pass/fail report to stdout. Something like:
```
FYN-LLM Fitness Suite
======================
Endpoint: http://10.0.0.10:3333/v1
Model: fyn-llm-agent-chat

[PASS] Mandelbrot reasoning     (1.2s, 340 tokens)
[PASS] JSON compliance (simple) (0.8s, 120 tokens)
[PASS] JSON compliance (nested) (0.9s, 280 tokens)
[FAIL] Instruction following: expected 3 bullets, got 5
[PASS] Summarization            (1.1s, 200 tokens)
...

Results: 8/10 passed
```

Exit code 0 if all pass, 1 if any fail.

### File Structure

```
backend/pipeline/quality/
  __init__.py          # Empty or minimal
  __main__.py          # argparse: subcommands fitness, (future: score, optimize)
  fitness.py           # FitnessRunner class with test methods
```

### Key Constraints

- **No new dependencies** — `openai`, `pydantic`, `argparse` are all already available
- **Runs outside Docker** — must work with `python -m pipeline.quality fitness` from `backend/` dir with appropriate env vars
- **Runs inside Docker** — should also work via `docker exec chrysopedia-api python -m pipeline.quality fitness`
- **Settings reuse** — use `config.get_settings()` for LLM endpoint config
- **Non-destructive** — fitness tests don't write to DB, don't trigger pipeline stages, don't modify any state

### Natural Task Seams

1. **CLI scaffold + basic connectivity test** — Create `quality/` package, `__main__.py` with argparse, `fitness.py` with a simple "can we reach the LLM" test. Verify `python -m pipeline.quality fitness` runs and prints something.

2. **Full test battery** — Implement all four test categories (Mandelbrot, JSON compliance, instruction following, diverse battery). Each test is a method on a `FitnessRunner` class. Print pass/fail report with timing.

3. **Polish + verification** — Exit codes, error handling (endpoint unreachable = clear error, not traceback), summary line, possibly `--verbose` flag for detailed output.

Tasks 1 and 2 could merge into a single task since the total code is modest (~200-300 lines). Task 3 is small cleanup.

### Verification

```bash
cd backend && python -m pipeline.quality fitness
```
Must produce pass/fail output for all test categories. Exit code 0 on all-pass, 1 on any failure. When run against an unreachable endpoint, should print a clear connectivity error rather than a raw traceback.

### Risks

- **LLM endpoint availability** — Tests hit a live LLM. If the endpoint is down, all tests fail. The CLI should catch connection errors and report "endpoint unreachable" clearly.
- **Model behavior variability** — Even at temperature 0.0, model outputs aren't perfectly deterministic across restarts. Pass/fail criteria should be generous (check for key content, not exact strings).
- **Mandelbrot validation** — Need to define what constitutes a "correct" answer without being too brittle. Check for presence of key concepts rather than exact phrasing.
