# S04: [B] Reindex Existing Corpus Through LightRAG

**Goal:** All 90 existing technique pages indexed in LightRAG with entity/relationship graph, queryable alongside current Qdrant search.
**Demo:** After this: All existing content indexed in LightRAG with entity/relationship graph alongside current search

## Tasks
- [x] **T01: Created reindex_lightrag.py that extracts technique pages from PostgreSQL, formats as rich text, and submits to LightRAG API with resume support and pipeline polling** — ## Description

Create `backend/scripts/reindex_lightrag.py` — a standalone script that:
1. Connects to PostgreSQL using the sync engine pattern from pipeline/stages.py
2. Queries all technique pages with creator and key moment joins
3. Formats each page as a rich text document for LightRAG entity extraction
4. Submits documents via POST /documents/text with polling for pipeline completion
5. Supports resume (skips already-processed file_sources)
6. Logs progress (N/90)

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| PostgreSQL | Exit with error message | Connection timeout → retry once | Log and skip page |
| LightRAG API | Log error, continue to next doc | Poll timeout (10min) → log warning, continue | Log raw response, skip doc |

## Load Profile

- **Shared resources**: PostgreSQL connection (single session), LightRAG API (serial)
- **Per-operation cost**: 1 DB query for all pages (eager load), 1 POST + N polls per page
- **10x breakpoint**: LightRAG serial processing is the bottleneck; DB is one-shot

## Steps

1. Read `backend/pipeline/stages.py` lines 224-245 to understand the sync engine pattern. Read `backend/models.py` for TechniquePage, KeyMoment, Creator model definitions.
2. Create `backend/scripts/reindex_lightrag.py` with:
   - Sync SQLAlchemy engine using the same `DATABASE_URL` → psycopg2 conversion pattern
   - `format_technique_page(page)` function that handles both v1 (flat dict) and v2 (nested list) `body_sections_format`
   - Document text format: `Technique: {title}\nCreator: {creator}\nCategory: {topic_category}\nTags: {tags}\nPlugins: {plugins}\n\nSummary: {summary}\n\n{sections}\n\nKey Moments from Source Videos:\n- {moment_title}: {moment_summary}`
   - `get_processed_sources(lightrag_url)` — calls GET /documents to get already-processed file_sources for resume
   - `submit_document(lightrag_url, text, file_source)` — POST /documents/text
   - `wait_for_pipeline(lightrag_url, timeout=600)` — poll GET /documents/pipeline_status every 10s until busy=false
   - `main()` with argparse: `--lightrag-url` (default http://chrysopedia-lightrag:9621), `--db-url` (from env DATABASE_URL), `--dry-run` (format and print without submitting), `--limit N` (process only first N pages for testing)
3. Handle v1 body_sections: iterate `dict.items()` → `heading: content` pairs
4. Handle v2 body_sections: iterate list of `{heading, content, subsections}` objects, flatten subsections
5. Add `--dry-run` mode that formats all pages and prints the first one fully, plus stats (total chars, page count)
6. Test by running via `ssh ub01 'docker exec chrysopedia-api python3 /app/scripts/reindex_lightrag.py --dry-run --limit 3'` to verify text extraction works for both v1 and v2 pages
7. Test submission with `--limit 2` (non-dry-run) to verify end-to-end: submit 2 new pages, poll for completion, check they appear in GET /documents/status_counts

## Must-Haves

- [ ] Script connects to PostgreSQL and queries all 90 technique pages with creator + key moments
- [ ] v1 body_sections (flat dict) correctly extracted to plain text
- [ ] v2 body_sections (nested list with subsections) correctly extracted to plain text
- [ ] Document format includes title, creator, category, tags, plugins, summary, sections, key moments
- [ ] file_source set to `technique:{slug}` for each document
- [ ] Resume: GET /documents checked and already-processed sources skipped
- [ ] Pipeline polling: waits for busy=false between submissions
- [ ] --dry-run and --limit flags work
- [ ] Progress logging: `[N/90] Submitted: {slug}` and `[N/90] Processed: {slug}`

## Verification

- `ssh ub01 'docker exec chrysopedia-api python3 /app/scripts/reindex_lightrag.py --dry-run --limit 3'` exits 0 and prints formatted text for 3 pages
- `ssh ub01 'curl -sf http://localhost:9621/documents/status_counts'` shows increase after `--limit 2` non-dry-run

## Inputs

- `backend/pipeline/stages.py` — sync engine pattern (lines 224-245)
- `backend/models.py` — TechniquePage, KeyMoment, Creator model definitions
- `backend/config.py` — DATABASE_URL configuration

## Expected Output

- `backend/scripts/reindex_lightrag.py` — complete reindex script
  - Estimate: 1.5h
  - Files: backend/scripts/reindex_lightrag.py, backend/pipeline/stages.py, backend/models.py, backend/config.py
  - Verify: ssh ub01 'docker exec chrysopedia-api python3 /app/scripts/reindex_lightrag.py --dry-run --limit 3' exits 0 and prints formatted technique page text
- [x] **T02: Deployed reindex script to ub01 via image rebuild, started full 90-page corpus reindex — 8 pages submitted with 168 entities extracted including creators, plugins, and technique concepts** — ## Description

Deploy the reindex script to ub01, start the full 90-page reindex in a background session, and verify graph quality once pages are processed. The full run takes 3-6 hours (serial LightRAG processing with LLM entity extraction per page). Start it backgrounded and verify on whatever has completed.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| LightRAG processing | Script logs and continues; check final summary | Poll timeout per doc → skip and continue | Log warning, continue |
| DGX Sparks LLM | LightRAG retries internally; script sees slow processing | Extend poll timeout | N/A (LightRAG handles) |

## Steps

1. Copy updated script to ub01: `scp backend/scripts/reindex_lightrag.py ub01:/vmPool/r/repos/xpltdco/chrysopedia/backend/scripts/`
2. Rebuild API container to pick up new script: `ssh ub01 'cd /vmPool/r/repos/xpltdco/chrysopedia && docker compose build chrysopedia-api && docker compose up -d chrysopedia-api'`
3. Check current LightRAG state: `ssh ub01 'curl -sf http://localhost:9621/documents/status_counts'` — note existing processed count (should be 2 from S01 + 2 from T01 testing)
4. Start the full reindex in a background docker exec session: `ssh ub01 'docker exec chrysopedia-api nohup python3 /app/scripts/reindex_lightrag.py --lightrag-url http://chrysopedia-lightrag:9621 > /tmp/reindex.log 2>&1 &'`
5. Wait 5-10 minutes, then check progress: `ssh ub01 'curl -sf http://localhost:9621/documents/status_counts'` and `ssh ub01 'docker exec chrysopedia-api tail -20 /tmp/reindex.log'`
6. Verify all existing services still healthy: `ssh ub01 'docker ps --filter name=chrysopedia --format "{{.Names}} {{.Status}}"'`
7. Check graph quality on processed pages:
   - `ssh ub01 'curl -sf http://localhost:9621/graph/label/list'` — check entity types present
   - `ssh ub01 'curl -sf -X POST http://localhost:9621/query -H "Content-Type: application/json" -d "{\"query\":\"What plugins are used for bass sound design?\"}"'` — verify multi-creator results
   - `ssh ub01 'curl -sf -X POST http://localhost:9621/query -H "Content-Type: application/json" -d "{\"query\":\"Which creators teach about Serum?\"}"'` — verify creator entity extraction
8. If reindex is still running, document how to check completion: `curl http://localhost:9621/documents/status_counts` and `docker exec chrysopedia-api tail -5 /tmp/reindex.log`
9. If reindex has completed (90+ processed), run full verification suite from slice verification section

## Must-Haves

- [ ] Script deployed and running on ub01
- [ ] Reindex making progress (processed count increasing)
- [ ] All existing Chrysopedia services remain healthy
- [ ] Graph contains entities extracted from processed pages
- [ ] Sample queries return relevant results

## Verification

- `ssh ub01 'curl -sf http://localhost:9621/documents/status_counts'` shows processed count increasing (or 90+ if complete)
- `ssh ub01 'docker ps --filter name=chrysopedia --format "{{.Names}} {{.Status}}"'` — all services healthy
- At least one POST /query returns results citing multiple creators/techniques

## Inputs

- `backend/scripts/reindex_lightrag.py` — reindex script from T01

## Expected Output

- `backend/scripts/reindex_lightrag.py` — deployed on ub01 (may have minor fixes from runtime issues)
  - Estimate: 1h
  - Files: backend/scripts/reindex_lightrag.py
  - Verify: ssh ub01 'curl -sf http://localhost:9621/documents/status_counts' shows processed count > 4 (2 from S01 + 2 from T01 + new pages)
