# S07 Research: Search — Per-Section Embeddings + Deep Linking

## Summary

This slice adds section-level search granularity and deep linking. Currently, stage 6 embeds whole technique pages and key moments as two point types in Qdrant. S07 adds a third point type (`technique_section`) that embeds individual v2 body_sections, enabling search results that link directly to a specific section within a technique page (e.g., `/techniques/lfo-modulation#grain-position-control`). The frontend already has section IDs and `scroll-margin-top` from S05 but lacks hash-based scroll handling — React Router client-side navigation doesn't auto-scroll to anchors.

Straightforward extension of established patterns. No new technology, no ambiguous requirements.

## Recommendation

Three tasks:

1. **Backend: Section embedding in stage 6 + new Qdrant upsert method** — Extend `stage6_embed_and_index` to iterate v2 `body_sections`, embed each section (heading + content + subsection content), and upsert to Qdrant with type `technique_section`. Add `upsert_technique_sections` to `QdrantManager`. Extend `SearchService._enrich_qdrant_results` to handle the new type. Update `SearchResultItem` schema with `section_anchor` field.

2. **Frontend: Hash scroll handler + section result rendering** — Add `useHashScroll` hook to TechniquePage that scrolls to the hash anchor after content loads. Update `SearchResultItem` type with `section_anchor`. Update `getSearchResultLink` and `SearchResultCard` to show section context and link to `page#section-slug`.

3. **Verification: End-to-end search → deep link test** — Verify stage 6 produces section points in Qdrant, search returns them with section_anchor, and clicking a result navigates to the correct section.

## Implementation Landscape

### Current Indexing Pipeline (stage 6)

**File:** `backend/pipeline/stages.py` lines 1595–1762

Stage 6 loads technique pages and key moments for a video, embeds text via `EmbeddingClient`, and upserts to Qdrant via `QdrantManager`. Two point types exist:

- `technique_page` — embed text: `"{creator} {title} {category} {tags} {summary}"`, payload: page_id, creator_id, creator_name, title, slug, topic_category, topic_tags, summary
- `key_moment` — embed text: `"{creator} {title} {summary}"`, payload: moment_id, source_video_id, technique_page_id, technique_page_slug, title, creator_name, timestamps, content_type

Deterministic UUIDs via `uuid5(_QDRANT_NAMESPACE, "tp:{page_id}")` / `"km:{moment_id}"`.

### QdrantManager

**File:** `backend/pipeline/qdrant_client.py`

Has `upsert_technique_pages` and `upsert_key_moments` — both build `PointStruct` lists and call `upsert_points`. Adding `upsert_technique_sections` follows the identical pattern. Deterministic UUID would be `uuid5(_QDRANT_NAMESPACE, "ts:{page_id}:{section_slug}")`.

### Search Service

**File:** `backend/search_service.py`

`search_qdrant()` returns payload dicts. `_enrich_qdrant_results()` maps payload fields to `SearchResultItem`-shaped dicts based on `payload["type"]`. Currently handles `technique_page` and falls through for `key_moment`. Needs a `technique_section` branch that maps `section_heading`, `section_anchor`, and parent page slug.

Keyword search (`_keyword_search_and`) queries PostgreSQL directly — no section-level keyword search needed since `body_sections` is a JSONB column and full-text indexing it would be complex with diminishing returns. Semantic search is the right tool for section-level granularity.

### Search API Schema

**File:** `backend/schemas.py`

`SearchResultItem` needs a new optional field: `section_anchor: str = ""`. This carries the anchor slug (e.g., `grain-position-control`) so the frontend can build `page#anchor` links.

### Frontend Search

**File:** `frontend/src/pages/SearchResults.tsx`

`getSearchResultLink()` already builds hash links for key_moments (`#km-{slug}`). Adding section deep links: if `item.type === "technique_section"`, return `/techniques/${item.technique_page_slug}#${item.section_anchor}`.

**File:** `frontend/src/api/public-client.ts`

`SearchResultItem` interface needs `section_anchor?: string`.

### Frontend Hash Scrolling (GAP)

**File:** `frontend/src/pages/TechniquePage.tsx`

Sections already have `id={sectionSlug}` and `scroll-margin-top: 5rem` CSS. But there is NO hash scroll handling — React Router client-side navigation won't auto-scroll to `#anchor`. Need a `useEffect` that reads `location.hash` after technique data loads and calls `document.getElementById(hash)?.scrollIntoView({ behavior: 'smooth' })`. Must run after render (when section elements exist in DOM).

### Section ID Convention (from S05)

`slugify()` in `frontend/src/components/TableOfContents.tsx`: lowercase, replace non-alphanumeric with hyphens, trim leading/trailing hyphens.

Section IDs: `slugify(section.heading)` → e.g., `grain-position-control`
Subsection IDs: `${sectionSlug}--${slugify(sub.heading)}` → e.g., `grain-position-control--lfo-routing`

The backend must replicate this slugify logic in Python for the `section_anchor` payload field. Python equivalent: `re.sub(r'[^a-z0-9]+', '-', heading.lower()).strip('-')`.

### v2 Body Sections Structure

Stored as JSONB in `TechniquePage.body_sections`. Format when `body_sections_format='v2'`:

```json
[
  {
    "heading": "Section Title",
    "content": "Section prose...",
    "subsections": [
      { "heading": "Sub Title", "content": "Sub prose..." }
    ]
  }
]
```

Per D024: sections with subsections have empty-string content; substance lives in subsection content fields.

### Embedding Text Construction for Sections

For each section, concatenate: `"{creator_name} {page_title} — {section_heading}: {section_content} {subsection_contents}"`. This gives the embedding model context about what page/creator the section belongs to while focusing on the section's specific content.

### Volume Estimate

Typical technique pages have 3-7 sections. With ~N pages in the system, this adds ~4N section points to Qdrant. The existing collection handles this without configuration changes — same vector dimensions, same distance metric.

### Deduplication on Re-index

Deterministic UUID `uuid5(namespace, "ts:{page_id}:{section_slug}")` ensures re-indexing the same page overwrites the same section points. However, if a compose operation changes section headings, orphan points from the old heading remain. Acceptable for now — a full re-index (`stage6_embed_and_index`) replaces all points for a video's pages anyway.

### Risk: Section heading changes after compose

When a page is composed with new video content, section headings may change. The old section points in Qdrant won't be deleted automatically. Mitigation: before upserting section points for a page, delete existing `technique_section` points with that `page_id`. This requires a `QdrantManager.delete_sections_by_page_id()` method using a filter on `payload.page_id`.

## Constraints

- Section embeddings only apply to v2 pages (`body_sections_format='v2'`). v1 pages continue with page-level embedding only.
- The `slugify` function must be identical in Python (stage 6) and TypeScript (frontend) — any divergence breaks deep links.
- Hash scrolling must wait for the technique page data to load and render before scrolling, otherwise the target element doesn't exist yet.
