---
estimated_steps: 27
estimated_files: 5
skills_used: []
---

# T01: Backend — Section embedding, Qdrant upsert, search enrichment, schema

Add per-section embedding to stage 6 for v2 technique pages. Extend QdrantManager with upsert_technique_sections() and delete_sections_by_page_id(). Extend SearchService._enrich_qdrant_results() with technique_section handling. Add section_anchor and section_heading to SearchResultItem schema. Write unit tests.

## Steps

1. **Add `section_anchor` and `section_heading` to SearchResultItem** in `backend/schemas.py`. Both optional strings defaulting to empty.

2. **Add `_slugify_heading()` helper** to `backend/pipeline/stages.py` — Python equivalent of the frontend's slugify: `re.sub(r'[^a-z0-9]+', '-', heading.lower()).strip('-')`. This must produce identical output to `frontend/src/components/TableOfContents.tsx` slugify.

3. **Add `delete_sections_by_page_id()` to QdrantManager** in `backend/pipeline/qdrant_client.py`. Uses a Qdrant filter on `payload.page_id` AND `payload.type == 'technique_section'` to delete stale section points before re-upserting. Non-blocking (logs warning on failure).

4. **Add `upsert_technique_sections()` to QdrantManager** — follows the same pattern as `upsert_technique_pages()`. Deterministic UUID: `uuid5(_QDRANT_NAMESPACE, f'ts:{page_id}:{section_slug}')`. Payload includes: type='technique_section', page_id, creator_id, creator_name, title (page title), slug (page slug), section_heading, section_anchor, topic_category, topic_tags, summary (section content truncated to 200 chars).

5. **Extend stage 6 to embed v2 sections**. After the existing technique page embedding block, add a new block that iterates pages where `body_sections_format == 'v2'` and `body_sections` is a list. For each section: build embed text as `'{creator_name} {page_title} — {section_heading}: {section_content} {subsection_contents}'`, build metadata dict with section_anchor from `_slugify_heading(section.heading)`. Before upserting, call `delete_sections_by_page_id()` for each page to remove stale points. Then batch embed and upsert. Log count of section points.

6. **Add `technique_section` branch to `_enrich_qdrant_results()`** in `backend/search_service.py`. Map payload fields: type='technique_section', title=section_heading, slug=page_slug, technique_page_slug=page_slug, section_anchor from payload, section_heading from payload. Creator resolution same as other types.

7. **Write unit tests** in `backend/pipeline/test_section_embedding.py`. Test: (a) _slugify_heading produces correct slugs for various headings, (b) section embed text construction includes creator/page/section context, (c) delete_sections_by_page_id is called before upsert, (d) v1 pages are skipped, (e) upsert_technique_sections builds correct payloads with deterministic UUIDs.

## Must-Haves

- [ ] _slugify_heading output matches frontend slugify for: 'Grain Position Control' → 'grain-position-control', 'LFO Routing & Modulation' → 'lfo-routing---modulation' (consecutive special chars become single hyphen — verify actual frontend behavior)
- [ ] Deterministic UUIDs: same page+section always overwrites the same Qdrant point
- [ ] delete_sections_by_page_id called before upsert to prevent orphan points from heading renames
- [ ] v1 pages (body_sections_format != 'v2') produce zero section points
- [ ] SearchResultItem.section_anchor populated for technique_section results
- [ ] Existing technique_page and key_moment enrichment unchanged

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| Qdrant | Log WARNING, skip section upsert (non-blocking) | Same — stage 6 is best-effort | Skip point, continue |
| Embedding API | Log WARNING, skip section embedding | Same | Skip batch, continue |
| body_sections JSONB | Skip page if not a list or missing heading field | N/A | Log WARNING, skip malformed section |

## Negative Tests

- Page with body_sections_format='v1' → zero section points
- Page with body_sections=None → zero section points
- Section with empty heading → skipped
- Section with subsections but no direct content → subsection content still embedded

## Inputs

- ``backend/pipeline/stages.py` — existing stage6_embed_and_index function`
- ``backend/pipeline/qdrant_client.py` — existing QdrantManager with upsert_technique_pages pattern`
- ``backend/search_service.py` — existing _enrich_qdrant_results with technique_page/key_moment handling`
- ``backend/schemas.py` — existing SearchResultItem class`
- ``frontend/src/components/TableOfContents.tsx` — reference slugify implementation (lines 10-15) to match in Python`

## Expected Output

- ``backend/schemas.py` — SearchResultItem with section_anchor and section_heading fields`
- ``backend/pipeline/stages.py` — _slugify_heading helper + section embedding block in stage6_embed_and_index`
- ``backend/pipeline/qdrant_client.py` — upsert_technique_sections() and delete_sections_by_page_id() methods`
- ``backend/search_service.py` — technique_section branch in _enrich_qdrant_results`
- ``backend/pipeline/test_section_embedding.py` — unit tests for section embedding pipeline`

## Verification

PYTHONPATH=backend python -m pytest backend/pipeline/test_section_embedding.py -v && PYTHONPATH=backend python -c "from pipeline.stages import _slugify_heading; assert _slugify_heading('Grain Position Control') == 'grain-position-control'; print('slugify OK')" && grep -q 'section_anchor' backend/schemas.py && grep -q 'technique_section' backend/search_service.py
