---
estimated_steps: 34
estimated_files: 1
skills_used: []
---

# T02: Run full reindex on ub01 and verify graph quality

## Description

Deploy the reindex script to ub01, start the full 90-page reindex in a background session, and verify graph quality once pages are processed. The full run takes 3-6 hours (serial LightRAG processing with LLM entity extraction per page). Start it backgrounded and verify on whatever has completed.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| LightRAG processing | Script logs and continues; check final summary | Poll timeout per doc → skip and continue | Log warning, continue |
| DGX Sparks LLM | LightRAG retries internally; script sees slow processing | Extend poll timeout | N/A (LightRAG handles) |

## Steps

1. Copy updated script to ub01: `scp backend/scripts/reindex_lightrag.py ub01:/vmPool/r/repos/xpltdco/chrysopedia/backend/scripts/`
2. Rebuild API container to pick up new script: `ssh ub01 'cd /vmPool/r/repos/xpltdco/chrysopedia && docker compose build chrysopedia-api && docker compose up -d chrysopedia-api'`
3. Check current LightRAG state: `ssh ub01 'curl -sf http://localhost:9621/documents/status_counts'` — note existing processed count (should be 2 from S01 + 2 from T01 testing)
4. Start the full reindex in a background docker exec session: `ssh ub01 'docker exec chrysopedia-api nohup python3 /app/scripts/reindex_lightrag.py --lightrag-url http://chrysopedia-lightrag:9621 > /tmp/reindex.log 2>&1 &'`
5. Wait 5-10 minutes, then check progress: `ssh ub01 'curl -sf http://localhost:9621/documents/status_counts'` and `ssh ub01 'docker exec chrysopedia-api tail -20 /tmp/reindex.log'`
6. Verify all existing services still healthy: `ssh ub01 'docker ps --filter name=chrysopedia --format "{{.Names}} {{.Status}}"'`
7. Check graph quality on processed pages:
   - `ssh ub01 'curl -sf http://localhost:9621/graph/label/list'` — check entity types present
   - `ssh ub01 'curl -sf -X POST http://localhost:9621/query -H "Content-Type: application/json" -d "{\"query\":\"What plugins are used for bass sound design?\"}"'` — verify multi-creator results
   - `ssh ub01 'curl -sf -X POST http://localhost:9621/query -H "Content-Type: application/json" -d "{\"query\":\"Which creators teach about Serum?\"}"'` — verify creator entity extraction
8. If reindex is still running, document how to check completion: `curl http://localhost:9621/documents/status_counts` and `docker exec chrysopedia-api tail -5 /tmp/reindex.log`
9. If reindex has completed (90+ processed), run full verification suite from slice verification section

## Must-Haves

- [ ] Script deployed and running on ub01
- [ ] Reindex making progress (processed count increasing)
- [ ] All existing Chrysopedia services remain healthy
- [ ] Graph contains entities extracted from processed pages
- [ ] Sample queries return relevant results

## Verification

- `ssh ub01 'curl -sf http://localhost:9621/documents/status_counts'` shows processed count increasing (or 90+ if complete)
- `ssh ub01 'docker ps --filter name=chrysopedia --format "{{.Names}} {{.Status}}"'` — all services healthy
- At least one POST /query returns results citing multiple creators/techniques

## Inputs

- `backend/scripts/reindex_lightrag.py` — reindex script from T01

## Expected Output

- `backend/scripts/reindex_lightrag.py` — deployed on ub01 (may have minor fixes from runtime issues)

## Inputs

- ``backend/scripts/reindex_lightrag.py` — complete reindex script from T01`

## Expected Output

- ``backend/scripts/reindex_lightrag.py` — deployed and running on ub01`

## Verification

ssh ub01 'curl -sf http://localhost:9621/documents/status_counts' shows processed count > 4 (2 from S01 + 2 from T01 + new pages)
