---
estimated_steps: 68
estimated_files: 7
skills_used: []
---

# T01: Build async search service and all public API endpoints

## Description

Create the backend API surface for S05: the async search service (embedding + Qdrant), search endpoint, technique pages CRUD, topics hierarchy, and enhanced creators endpoint. This is the highest-risk task because it introduces async embedding/Qdrant clients for the FastAPI request path (existing ones are sync for Celery).

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| Embedding API (AsyncOpenAI) | Fall back to keyword-only search | 300ms timeout → keyword fallback | Return empty vectors → keyword fallback |
| Qdrant (AsyncQdrantClient) | Fall back to keyword-only search | 300ms timeout → keyword fallback | Log warning, return empty results → keyword fallback |
| PostgreSQL | Return 500 (standard FastAPI error handling) | Connection pool timeout → 500 | N/A (SQLAlchemy typed) |

## Load Profile

- **Shared resources**: AsyncQdrantClient connection pool, AsyncOpenAI HTTP pool, SQLAlchemy async session pool
- **Per-operation cost**: Search = 1 embedding API call + 1 Qdrant query + 1-3 SQL queries for enrichment. Read endpoints = 1-2 SQL queries each.
- **10x breakpoint**: Embedding API rate limiting (external dependency). Mitigated by client-side debounce (300ms) reducing request rate.

## Negative Tests

- **Malformed inputs**: Empty search query → return empty results. Query > 500 chars → truncate to 500. Invalid scope parameter → default to 'all'.
- **Error paths**: Embedding API unreachable → keyword fallback. Qdrant unreachable → keyword fallback. Invalid slug → 404.
- **Boundary conditions**: Empty Qdrant collection → keyword-only results. Zero matching techniques/creators → empty list.

## Steps

1. Create `backend/search_service.py` with `SearchService` class:
   - `__init__` takes Settings, creates `openai.AsyncOpenAI` client and `qdrant_client.AsyncQdrantClient`
   - `async embed_query(text: str) -> list[float] | None` — embeds query text with 300ms timeout, returns None on failure
   - `async search_qdrant(vector: list[float], limit: int, type_filter: str | None) -> list[dict]` — queries Qdrant with optional payload type filter, returns scored results with payloads
   - `async keyword_search(query: str, scope: str, limit: int, db: AsyncSession) -> list[dict]` — ILIKE search on technique_pages.title, key_moments.title, creators.name
   - `async search(query: str, scope: str, limit: int, db: AsyncSession) -> dict` — orchestrates: embed → Qdrant → enrich with DB metadata → fallback to keyword if needed

2. Add new Pydantic response schemas to `backend/schemas.py`:
   - `SearchResultItem(title, slug, type, score, summary, creator_name, creator_slug, topic_category, topic_tags)`
   - `SearchResponse(items: list[SearchResultItem], total: int, query: str, fallback_used: bool)`
   - `TechniquePageDetail` (extends TechniquePageRead with nested key_moments, creator info, related links)
   - `TopicCategory(name, description, sub_topics: list[TopicSubTopic])` and `TopicSubTopic(name, technique_count, creator_count)`
   - `CreatorBrowseItem` (extends CreatorRead with technique_count, video_count)

3. Create `backend/routers/search.py`:
   - `GET /search?q=...&scope=all|topics|creators&limit=20`
   - Instantiate SearchService from get_settings(), call search(), return SearchResponse
   - Log query, latency_ms, result_count, fallback_used at INFO level

4. Create `backend/routers/techniques.py`:
   - `GET /techniques` — list technique pages with optional `category`, `creator_slug` query filters, pagination
   - `GET /techniques/{slug}` — full detail with eager-loaded key_moments (ordered by start_time), creator info, outgoing+incoming related links
   - Return 404 for unknown slug

5. Create `backend/routers/topics.py`:
   - `GET /topics` — load `canonical_tags.yaml`, for each category aggregate technique_count and creator_count per sub_topic from DB
   - `GET /topics/{category_slug}` — return technique pages filtered by topic_category

6. Extend `backend/routers/creators.py`:
   - Add `sort` query param: `random` (default), `alpha`, `views`
   - Add `genre` query param for filtering by genre
   - Add technique_count and video_count subqueries to list endpoint
   - For `sort=random`, use `func.random()` ORDER BY (dataset is small, <100 creators)

7. Mount all new routers in `backend/main.py`:
   - `from routers import search, techniques, topics`
   - `app.include_router(search.router, prefix="/api/v1")`
   - `app.include_router(techniques.router, prefix="/api/v1")`
   - `app.include_router(topics.router, prefix="/api/v1")`

## Must-Haves

- [ ] SearchService with async embedding + Qdrant + keyword fallback
- [ ] GET /api/v1/search returns SearchResponse with enriched results
- [ ] GET /api/v1/techniques and GET /api/v1/techniques/{slug} with full detail
- [ ] GET /api/v1/topics returns category hierarchy with counts
- [ ] GET /api/v1/creators supports sort=random (default), genre filter, technique/video counts
- [ ] All new routers mounted in main.py
- [ ] Embedding/Qdrant failures gracefully degrade to keyword search

## Verification

- `cd backend && python -c "from search_service import SearchService; print('OK')"` — imports clean
- `cd backend && python -c "from routers.search import router; print(router.routes)"` — search router has routes
- `cd backend && python -c "from routers.techniques import router; print(router.routes)"` — techniques router has routes
- `cd backend && python -c "from routers.topics import router; print(router.routes)"` — topics router has routes
- `cd backend && python -c "from main import app; routes = [r.path for r in app.routes]; assert '/api/v1/search' in str(routes) or any('search' in str(r.path) for r in app.routes); print('Mounted')"` — routers mounted

## Observability Impact

- Signals added: INFO log per search query with latency_ms, result_count, fallback_used. WARNING on embedding/Qdrant failure with error details.
- How a future agent inspects this: `curl localhost:8001/api/v1/search?q=test` returns structured JSON with timing data
- Failure state exposed: fallback_used=true in search response indicates Qdrant/embedding degradation

## Inputs

- ``backend/models.py` — all 7 ORM models (Creator, SourceVideo, TranscriptSegment, KeyMoment, TechniquePage, RelatedTechniqueLink, Tag)`
- ``backend/database.py` — async engine, get_session dependency`
- ``backend/config.py` — Settings with embedding_api_url, embedding_model, embedding_dimensions, qdrant_url, qdrant_collection`
- ``backend/schemas.py` — existing Pydantic schemas to extend`
- ``backend/routers/creators.py` — existing creators router to enhance`
- ``backend/main.py` — existing router mounting to extend`
- ``backend/pipeline/embedding_client.py` — reference for sync embedding pattern (async variant needed)`
- ``backend/pipeline/qdrant_client.py` — reference for sync Qdrant pattern (async variant needed)`
- ``config/canonical_tags.yaml` — tag taxonomy for topics endpoint`

## Expected Output

- ``backend/search_service.py` — async SearchService with embed_query, search_qdrant, keyword_search, search methods`
- ``backend/schemas.py` — extended with SearchResultItem, SearchResponse, TechniquePageDetail, TopicCategory, TopicSubTopic, CreatorBrowseItem`
- ``backend/routers/search.py` — GET /search endpoint with semantic + keyword fallback`
- ``backend/routers/techniques.py` — GET /techniques and GET /techniques/{slug} endpoints`
- ``backend/routers/topics.py` — GET /topics endpoint with category hierarchy`
- ``backend/routers/creators.py` — enhanced with sort=random, genre filter, counts`
- ``backend/main.py` — all new routers mounted`

## Verification

cd backend && python -c "from search_service import SearchService; from routers.search import router as sr; from routers.techniques import router as tr; from routers.topics import router as tpr; print('All imports OK')" && python -c "from main import app; print([r.path for r in app.routes])"
