---
estimated_steps: 52
estimated_files: 6
skills_used: []
---

# T01: Config extensions, Celery app, LLM client, and pipeline schemas

Extend Settings with LLM/embedding/Qdrant/prompt config vars, add openai and qdrant-client to requirements.txt, create the Celery app in worker.py, build the sync LLM client with primary/fallback logic, and define Pydantic schemas for all pipeline stage inputs/outputs. This task establishes all infrastructure the pipeline stages depend on.

## Steps

1. Add `openai>=1.0,<2.0`, `qdrant-client>=1.9,<2.0`, and `pyyaml>=6.0,<7.0` to `backend/requirements.txt`.

2. Extend `backend/config.py` `Settings` class with these fields (all with sensible defaults from .env.example):
   - `llm_api_url: str = 'http://localhost:11434/v1'`
   - `llm_api_key: str = 'sk-placeholder'`
   - `llm_model: str = 'qwen2.5:14b-q8_0'`
   - `llm_fallback_url: str = 'http://localhost:11434/v1'`
   - `llm_fallback_model: str = 'qwen2.5:14b-q8_0'`
   - `embedding_api_url: str = 'http://localhost:11434/v1'`
   - `embedding_model: str = 'nomic-embed-text'`
   - `embedding_dimensions: int = 768`
   - `qdrant_url: str = 'http://localhost:6333'`
   - `qdrant_collection: str = 'chrysopedia'`
   - `prompts_path: str = './prompts'`
   - `review_mode: bool = True`

3. Create `backend/worker.py`: Celery app instance using `config.get_settings().redis_url` as broker. Import and register tasks from `pipeline.stages` (import the module so decorators register). Worker must be importable as `celery -A worker worker`.

4. Create `backend/pipeline/__init__.py` (empty).

5. Create `backend/pipeline/schemas.py` with Pydantic models:
   - `TopicSegment(start_index, end_index, topic_label, summary)` — stage 2 output per segment group
   - `SegmentationResult(segments: list[TopicSegment])` — stage 2 full output
   - `ExtractedMoment(title, summary, start_time, end_time, content_type, plugins, raw_transcript)` — stage 3 output per moment
   - `ExtractionResult(moments: list[ExtractedMoment])` — stage 3 full output
   - `ClassifiedMoment(moment_index, topic_category, topic_tags, content_type_override)` — stage 4 output per moment
   - `ClassificationResult(classifications: list[ClassifiedMoment])` — stage 4 full output
   - `SynthesizedPage(title, slug, topic_category, topic_tags, summary, body_sections, signal_chains, plugins, source_quality)` — stage 5 output
   - `SynthesisResult(pages: list[SynthesizedPage])` — stage 5 full output
   All fields should use appropriate types: `content_type` as str (enum value), `body_sections` as dict, `signal_chains` as list[dict], etc.

6. Create `backend/pipeline/llm_client.py`:
   - Class `LLMClient` initialized with `settings: Settings`
   - Uses sync `openai.OpenAI(base_url=settings.llm_api_url, api_key=settings.llm_api_key)`
   - Method `complete(system_prompt: str, user_prompt: str, response_model: type[BaseModel] | None = None) -> str` that:
     a. Tries primary endpoint with `response_format={'type': 'json_object'}` if response_model is provided
     b. On connection/timeout error, tries fallback endpoint (`openai.OpenAI(base_url=settings.llm_fallback_url)` with `settings.llm_fallback_model`)
     c. Returns the raw completion text
     d. Logs WARNING on fallback, ERROR on total failure
   - Method `parse_response(text: str, model: type[T]) -> T` that validates JSON via Pydantic `model_validate_json()` with error handling
   - Catch `openai.APIConnectionError`, `openai.APITimeoutError` for fallback trigger
   - IMPORTANT: Use sync `openai.OpenAI`, not async — Celery tasks run in sync context

## Must-Haves

- [ ] Settings has all 12 new fields with correct defaults
- [ ] `openai`, `qdrant-client`, `pyyaml` in requirements.txt
- [ ] `celery -A worker worker` is syntactically valid (worker.py creates Celery app and imports pipeline.stages)
- [ ] LLMClient uses sync openai.OpenAI, not AsyncOpenAI
- [ ] LLMClient has primary/fallback logic with proper exception handling
- [ ] All 8 Pydantic schema classes defined with correct field types

## Verification

- `cd backend && python -c "from config import Settings; s = Settings(); print(s.llm_api_url, s.qdrant_url, s.review_mode)"` prints defaults
- `cd backend && python -c "from pipeline.schemas import SegmentationResult, ExtractionResult, ClassificationResult, SynthesisResult; print('schemas ok')"` exits 0
- `cd backend && python -c "from pipeline.llm_client import LLMClient; print('client ok')"` exits 0
- `cd backend && python -c "from worker import celery_app; print(celery_app.main)"` exits 0
- `grep -q 'openai' backend/requirements.txt && grep -q 'qdrant-client' backend/requirements.txt`

## Inputs

- ``backend/config.py` — existing Settings class to extend`
- ``backend/requirements.txt` — existing dependencies to append to`
- ``backend/database.py` — provides engine/session patterns used by worker`
- ``.env.example` — reference for env var names and defaults`

## Expected Output

- ``backend/config.py` — Settings extended with 12 new LLM/embedding/Qdrant/prompt/review fields`
- ``backend/requirements.txt` — openai, qdrant-client, pyyaml added`
- ``backend/worker.py` — Celery app instance configured with Redis broker`
- ``backend/pipeline/__init__.py` — empty package init`
- ``backend/pipeline/schemas.py` — 8 Pydantic models for pipeline stage I/O`
- ``backend/pipeline/llm_client.py` — LLMClient with sync OpenAI primary/fallback`

## Verification

cd backend && python -c "from config import Settings; s = Settings(); print(s.llm_api_url, s.qdrant_url, s.review_mode)" && python -c "from pipeline.schemas import SegmentationResult, ExtractionResult, ClassificationResult, SynthesisResult; print('schemas ok')" && python -c "from pipeline.llm_client import LLMClient; print('client ok')" && python -c "from worker import celery_app; print(celery_app.main)"
