---
id: S04
parent: M019
milestone: M019
provides:
  - LightRAG corpus index (90 technique pages, entity/relationship graph)
  - backend/scripts/reindex_lightrag.py — reusable reindex script with resume support
requires:
  - slice: S01
    provides: LightRAG service running on port 9621 with REST API
affects:
  - S06
key_files:
  - backend/scripts/reindex_lightrag.py
key_decisions:
  - Used httpx instead of requests (not in container image)
  - file_source format: technique:{slug} for deterministic resume
  - Serial submission with pipeline polling between docs to avoid LLM overload
  - Deployed via image rebuild so script persists across container restarts
patterns_established:
  - Reindex script pattern: sync SQLAlchemy engine → format → serial POST with poll-for-completion between docs
  - Resume via file_source deduplication against GET /documents
observability_surfaces:
  - GET /documents/status_counts — track reindex progress
  - GET /graph/label/list — verify entity extraction quality
  - /tmp/reindex.log inside API container — script progress log
drill_down_paths:
  - .gsd/milestones/M019/slices/S04/tasks/T01-SUMMARY.md
  - .gsd/milestones/M019/slices/S04/tasks/T02-SUMMARY.md
duration: ""
verification_result: passed
completed_at: 2026-04-03T22:55:22.749Z
blocker_discovered: false
---

# S04: [B] Reindex Existing Corpus Through LightRAG

**Built reindex_lightrag.py script and deployed it to ub01, starting full 90-page corpus reindex through LightRAG with 168 entities extracted from initial 8 pages processed.**

## What Happened

Created backend/scripts/reindex_lightrag.py — a standalone script that connects to PostgreSQL via sync SQLAlchemy, queries all technique pages with eager-loaded Creator and KeyMoment relations, formats each as structured text (title, creator, category, tags, plugins, summary, body sections for both v1/v2 formats, key moments), and submits serially to LightRAG's POST /documents/text endpoint. Resume support fetches existing file_paths from GET /documents and skips already-processed technique:{slug} entries. Pipeline polling waits for busy=false between submissions to avoid overwhelming the LLM backend.

T01 built the script and validated it: dry-run with --limit 3 printed formatted text for 3 pages correctly, live submission with --limit 2 processed both pages through entity extraction (30 entities + 26 relationships per chunk). Used httpx instead of requests since the container image doesn't include requests.

T02 deployed via image rebuild (baked into Dockerfile so it persists across restarts), then started the full 90-page reindex backgrounded inside the API container. After ~10 minutes: 8 pages submitted, 6 processed, 2 processing. Graph shows 168 entities including creators (e.g., Ableton-related entities), plugins (Serum, Operator, etc.), and technique concepts (bass design, arrangement, atmospheric textures). Query endpoint times out during active indexing due to shared LLM backend — expected behavior, will work once indexing completes.

The full reindex takes 3-6 hours (serial LightRAG processing with LLM entity extraction per page). It's running backgrounded and can be monitored via `curl http://localhost:9621/documents/status_counts` and `docker exec chrysopedia-api tail -5 /tmp/reindex.log`.

## Verification

1. `ssh ub01 'docker exec chrysopedia-api python3 /app/scripts/reindex_lightrag.py --dry-run --limit 3'` — exits 0, prints formatted text for 3 pages ✅
2. `ssh ub01 'curl -sf http://localhost:9621/documents/status_counts'` — shows 8 total docs, 6 processed (>4 threshold) ✅
3. `ssh ub01 'docker ps --filter name=chrysopedia'` — all 10 services healthy ✅
4. `ssh ub01 'curl -sf http://localhost:9621/graph/label/list'` — 168 entities with creators, plugins, technique concepts ✅
5. Query endpoint — times out during active indexing (shared LLM contention), expected behavior ⏳

## Requirements Advanced

None.

## Requirements Validated

None.

## New Requirements Surfaced

None.

## Requirements Invalidated or Re-scoped

None.

## Deviations

Used httpx instead of requests (not available in container image). T01 initially deployed via docker cp; T02 fixed this by doing a proper image rebuild so the script persists across container restarts.

## Known Limitations

Full 90-page reindex takes 3-6 hours due to serial LightRAG processing with LLM entity extraction. Query endpoint unavailable during active indexing (shared LLM backend contention). Only 8/90 pages submitted at slice completion — reindex continues in background.

## Follow-ups

Monitor reindex completion: `ssh ub01 'curl -sf http://localhost:9621/documents/status_counts'` should show processed:90+. After completion, validate query quality with cross-creator queries. Consider parallel submission or batching if re-indexing needs to happen frequently.

## Files Created/Modified

- `backend/scripts/reindex_lightrag.py` — New reindex script: PostgreSQL → format → LightRAG POST with resume, polling, dry-run, and limit flags
