---
estimated_steps: 43
estimated_files: 3
skills_used: []
---

# T03: Qdrant integration and embedding client

Create the embedding client (sync OpenAI-compatible /v1/embeddings) and Qdrant client (collection management + upsert). Wire embedding generation into the pipeline after stage 5 synthesis, so technique page summaries and key moment summaries are embedded and upserted to Qdrant with metadata payloads.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| Embedding API | Log error, skip embedding for this batch, pipeline continues | Same as error | Validate vector dimensions match config, log mismatch |
| Qdrant server | Log error, skip upsert, pipeline continues (embeddings are not blocking) | Same as error | N/A |

## Steps

1. Create `backend/pipeline/embedding_client.py`:
   - Class `EmbeddingClient` initialized with `settings: Settings`
   - Uses sync `openai.OpenAI(base_url=settings.embedding_api_url, api_key=settings.llm_api_key)`
   - Method `embed(texts: list[str]) -> list[list[float]]` that calls `client.embeddings.create(model=settings.embedding_model, input=texts)` and returns the vector list
   - Handle `openai.APIConnectionError` gracefully — log and return empty list
   - Validate returned vector dimensions match `settings.embedding_dimensions`

2. Create `backend/pipeline/qdrant_client.py`:
   - Class `QdrantManager` initialized with `settings: Settings`
   - Uses sync `qdrant_client.QdrantClient(url=settings.qdrant_url)`
   - Method `ensure_collection()` — checks `collection_exists()`, creates with `VectorParams(size=settings.embedding_dimensions, distance=Distance.COSINE)` if not
   - Method `upsert_points(points: list[PointStruct])` — wraps `client.upsert()`
   - Method `upsert_technique_pages(pages: list[dict], vectors: list[list[float]])` — builds PointStruct list with payload (page_id, creator_id, title, topic_category, topic_tags, summary) and upserts
   - Method `upsert_key_moments(moments: list[dict], vectors: list[list[float]])` — builds PointStruct list with payload (moment_id, source_video_id, title, start_time, end_time, content_type) and upserts
   - Handle `qdrant_client.http.exceptions.UnexpectedResponse` for connection errors

3. Add a new Celery task `stage6_embed_and_index(video_id: str)` in `backend/pipeline/stages.py`:
   - Load all KeyMoments and TechniquePages created for this video
   - Build text strings for embedding (moment: title + summary; page: title + summary + topic_category)
   - Call EmbeddingClient.embed() to get vectors
   - Call QdrantManager.ensure_collection() then upsert_technique_pages() and upsert_key_moments()
   - This stage runs after stage 5 in the pipeline chain but does NOT update processing_status (it's a side-effect)
   - If embedding/Qdrant fails, log error but don't fail the pipeline — embeddings can be regenerated later

4. Update `run_pipeline` in `stages.py` to append `stage6_embed_and_index` to the chain.

## Must-Haves

- [ ] EmbeddingClient uses sync openai.OpenAI for /v1/embeddings calls
- [ ] QdrantManager creates collection only if not exists
- [ ] Qdrant points include metadata payloads (page_id/moment_id, creator_id, topic, timestamps)
- [ ] stage6 is non-blocking — embedding/Qdrant failures don't fail the pipeline
- [ ] Vector dimension from config, not hardcoded

## Verification

- `cd backend && python -c "from pipeline.embedding_client import EmbeddingClient; print('embed ok')"` exits 0
- `cd backend && python -c "from pipeline.qdrant_client import QdrantManager; print('qdrant ok')"` exits 0
- `cd backend && python -c "from pipeline.stages import stage6_embed_and_index; print('stage6 ok')"` exits 0

## Observability Impact

- Signals added: INFO log for embedding batch size and Qdrant upsert count, WARNING on embedding/Qdrant failures with error details
- How a future agent inspects this: check Qdrant dashboard at QDRANT_URL, query collection point count
- Failure state exposed: embedding failures logged but pipeline completes successfully

## Inputs

- ``backend/pipeline/stages.py` — pipeline stages from T02 to add stage6 and update run_pipeline`
- ``backend/config.py` — Settings with embedding_api_url, embedding_model, embedding_dimensions, qdrant_url, qdrant_collection from T01`
- ``backend/pipeline/schemas.py` — Pydantic schemas from T01`
- ``backend/models.py` — KeyMoment and TechniquePage ORM models`

## Expected Output

- ``backend/pipeline/embedding_client.py` — EmbeddingClient with sync OpenAI embeddings`
- ``backend/pipeline/qdrant_client.py` — QdrantManager with collection management and upsert`
- ``backend/pipeline/stages.py` — updated with stage6_embed_and_index task and run_pipeline chain`

## Verification

cd backend && python -c "from pipeline.embedding_client import EmbeddingClient; print('embed ok')" && python -c "from pipeline.qdrant_client import QdrantManager; print('qdrant ok')" && python -c "from pipeline.stages import stage6_embed_and_index; print('stage6 ok')"
