# KNOWLEDGE

## Healthcheck in slim Python Docker images: use os.kill(1,0) not pgrep

**Context:** The `python:3.x-slim` Docker image doesn't include `procps`, so `pgrep -f watcher.py` fails in healthcheck commands. This applies to any slim-based image where a healthcheck needs to verify the main process is alive.

**Fix:** Use `python -c "import os; os.kill(1, 0)"` as the healthcheck command. PID 1 is always the container's entrypoint process. `os.kill(pid, 0)` checks if the process exists without sending a signal — it raises `ProcessLookupError` if the process is dead, causing the healthcheck to fail. No external tools needed.

## SQLAlchemy column names that shadow ORM functions

**Context:** If a model has a column named `relationship` (or any other ORM function name like `query`, `metadata`), Python resolves the name to the column's `MappedColumn` descriptor within the class body. This causes `relationship(...)` calls below it to fail with "MappedColumn object is not callable".

**Fix:** Import with an alias: `from sqlalchemy.orm import relationship as sa_relationship` and use `sa_relationship(...)` throughout. Alternatively, rename the database column, but the spec defines `relationship` as the column name so we preserved it.

## Docker Compose variable interpolation and `:?` syntax

**Context:** `${VAR:?error message}` in docker-compose.yml makes `docker compose config` fail when the variable is unset, even if `env_file: required: false` is used. The `required: false` only controls whether the `.env` file must exist — it does NOT provide defaults for variables referenced with `:?`.

**Fix:** Use `${VAR:-default}` for variables that need a fallback, or ensure the variable is always set in the environment.

## Host port 8000 conflict with kerf-engine

**Context:** The kerf-engine Docker container (`kerf-engine-kerf-engine-1`) binds to `0.0.0.0:8000`. When testing the Chrysopedia API locally (outside Docker), curl requests to `localhost:8000` hit the kerf-engine container instead of the local uvicorn process, even when uvicorn binds to `127.0.0.1:8000`.

**Fix:** Use an alternate port (e.g., 8001) for local testing. In Docker Compose, chrysopedia-api maps `127.0.0.1:8000:8000` which is fine because it goes through Docker's port forwarding, but will conflict with kerf-engine if both stacks run simultaneously. Consider changing one project's external port mapping.

## asyncpg rejects timezone-aware datetimes for TIMESTAMP WITHOUT TIME ZONE columns

**Context:** SQLAlchemy models using `default=datetime.now(timezone.utc)` produce timezone-aware Python datetimes. When the PostgreSQL column type is `TIMESTAMP WITHOUT TIME ZONE` (the default), asyncpg raises `DataError: can't subtract offset-naive and offset-aware datetimes`. This only surfaces when running against a real PostgreSQL database — SQLite-based tests won't catch it.

**Fix:** Use `datetime.now(timezone.utc).replace(tzinfo=None)` in the `_now()` helper, or change the column type to `TIMESTAMP WITH TIME ZONE`. We chose the former since the existing schema uses naive timestamps.

## asyncpg NullPool required for pytest-asyncio integration tests

**Context:** When using a session-scoped SQLAlchemy async engine with asyncpg in pytest-asyncio tests, the connection pool reuses connections across fixtures and test functions. This causes `InterfaceError: cannot perform operation: another operation is in progress` because the ASGI test client's session holds a connection while cleanup/verification fixtures try to use the same pool.

**Fix:** Use `poolclass=NullPool` when creating the test engine. Each connection is created fresh and immediately closed, eliminating contention. Performance cost is negligible for test suites.

## Testing Celery tasks that use sync SQLAlchemy: patch module-level globals

**Context:** Pipeline stages in `pipeline/stages.py` create their own sync SQLAlchemy engine/session via module-level `_engine` and `_SessionLocal` globals (because Celery is sync, not async). Tests need to redirect these to the test database, but the engine is created lazily at module scope.

**Fix:** Patch the module globals directly: `unittest.mock.patch.object(stages, '_engine', test_engine)` and `unittest.mock.patch.object(stages, '_SessionLocal', test_session_factory)`. This redirects all DB access in stage functions to the test database without modifying production code.

## Lazy imports in FastAPI handlers defeat simple mock patching

**Context:** When a FastAPI handler imports a function lazily (inside the function body) like `from pipeline.stages import run_pipeline`, patching `routers.ingest.run_pipeline` has no effect because the name is re-bound on every call from the source module.

**Fix:** Patch at the source module: `unittest.mock.patch('pipeline.stages.run_pipeline')`. The lazy import will pick up the mock from the source module. This applies to any handler that uses lazy imports to avoid circular dependencies at module load time.

## Separate async/sync clients for FastAPI vs Celery

**Context:** The Chrysopedia backend has both sync Celery tasks (pipeline stages using `openai.OpenAI`, `QdrantClient`, sync SQLAlchemy) and async FastAPI handlers. Reusing sync clients in async handlers blocks the event loop; reusing async clients in Celery risks nested event loop errors.

**Fix:** Create separate client classes: `SearchService` (async, for FastAPI request path) wraps `openai.AsyncOpenAI` and `AsyncQdrantClient`. The pipeline's `EmbeddingClient` and `QdrantManager` (sync, for Celery) remain untouched. This doubles the client code surface but eliminates the async/sync mismatch class of bugs entirely.

## Mocking SearchService at the router dependency level for tests

**Context:** The search endpoint creates a `SearchService` instance internally. Testing search results with real embedding API and Qdrant is fragile (external dependencies). Mocking individual `openai.AsyncOpenAI` or `AsyncQdrantClient` is complex.

**Fix:** Mock `SearchService` at the router level by patching the service instance in the endpoint function. This gives full control over search results in tests without complex async mock setup. Used in `test_search.py` — mock returns canned `SearchResponse` dicts.

## Frontend detail page without a single-resource GET endpoint

**Context:** The review queue backend has `GET /review/queue` (list, paginated) but no `GET /review/moments/{id}` for fetching a single moment. The MomentDetail page needs to display one specific moment by ID from the URL params.

**Fix:** MomentDetail fetches the full queue with `limit=500` and filters client-side by ID. This works for a single-admin tool with small datasets but would not scale. If the moment count grows significantly, add a dedicated `GET /review/moments/{id}` endpoint to the review router.

**Update (M004/S01):** Fixed properly — added `GET /review/moments/{id}` endpoint. Always add single-resource GET endpoints alongside list endpoints.

## CSS custom property count estimation

**Context:** When planning a full color abstraction pass (replacing all hardcoded hex/rgba with CSS custom properties), initial estimates based on counting unique colors will significantly undercount the required tokens.

**Rule:** Plan for 2-3x the obvious color count. Reasons: hover/focus/active state variants, badge/status color pairs (background + text), subtle vs strong accent variants, shadow colors, overlay backgrounds, and distinct semantic contexts (surface vs input vs card). M004/S02 planned ~30, needed 77.

## Chained selectinload for cross-relation field population

**Context:** When an API response needs a field from a joined relation (e.g., `key_moment.source_video.filename`), Pydantic's `from_attributes` auto-mapping can't traverse the join. The field appears as a flat attribute on the response schema but comes from a different table.

**Fix:** Chain `.selectinload()` in the query (e.g., `.selectinload(TechniquePage.key_moments).selectinload(KeyMoment.source_video)`), then post-process in a loop to copy the joined value into the schema field. This keeps the query efficient (two SELECTs) and the response schema flat.

## Pre-overwrite snapshot pattern for article versioning

**Context:** Need to track content versions without complex CDC (change data capture) infrastructure. The content lives in a single mutable row that gets overwritten on each pipeline run.

**Pattern:** Before mutating the row, SELECT its current state, serialize relevant fields to JSONB (`content_snapshot`), attach pipeline metadata (prompt file hashes, model config) as a second JSONB column, and INSERT into a versions table. Use best-effort error handling — snapshot failure logs ERROR but doesn't block the primary write. Composite unique index on (entity_id, version_number) enforces ordering at DB level.

## Stage 4 classification data stored in Redis (not DB columns)

**Context:** The KeyMoment SQLAlchemy model doesn't have `topic_tags` or `topic_category` columns. Stage 4 classification needs somewhere to store per-moment tag assignments that stage 5 can read.

**Fix:** Store classification results in Redis under key `chrysopedia:classification:{video_id}` with a 24-hour TTL. Stage 5 reads from Redis. This avoids schema migrations during initial pipeline development. The data is ephemeral — if Redis loses it, re-running stage 4 regenerates it.

## QdrantManager uses random UUIDs for point IDs

**Context:** `QdrantManager.upsert_technique_pages()` and `upsert_key_moments()` generate `uuid4()` for each Qdrant point. Re-indexing the same content creates duplicate points rather than updating existing ones.

**Fix (deferred):** Use deterministic UUIDs based on content hash (e.g., `uuid5(NAMESPACE, f"{technique_slug}:{section}")`) so re-indexing overwrites the same points. This should be addressed before running the pipeline on production data to avoid index bloat.

## Non-blocking side-effect pattern for external service calls in pipelines

**Context:** Celery pipeline stages that call external services (embedding API, Qdrant) should not fail the entire pipeline if those services are down. Stage 6 (embed_and_index) is valuable but not critical — the pipeline's primary output (technique pages in PostgreSQL) doesn't depend on it.

**Fix:** Use `max_retries=0` and a catch-all exception handler that logs WARNING and returns without raising. The pipeline orchestrator chains stage 6 after stage 5 but a failure there doesn't prevent `processing_status` from reaching its final state. This pattern applies to any "best-effort enrichment" stage in a pipeline.

## Container healthcheck tool availability varies by base image

**Context:** Qdrant (distroless-ish), Ollama, and nginx-alpine images don't have curl or wget. Standard healthcheck patterns like `curl -f http://localhost:PORT/health` fail silently inside these containers.

**Fix by image:**
- **Ollama:** Use `ollama list` (built-in CLI)
- **Qdrant:** Use `bash -c 'echo > /dev/tcp/localhost/6333'` (bash is available)
- **nginx-alpine:** Use `curl` (available) but NOT busybox `wget` (fails on localhost)
- **Celery worker (same image as API):** Use `celery -A worker inspect ping` instead of HTTP healthcheck

**Rule:** Always `docker exec <container> <healthcheck-cmd>` to verify the command works before deploying.

## XPLTD domain setup flow

**Context:** Adding a new external domain (like chrysopedia.com) to the XPLTD infrastructure requires three steps across two machines.

**Flow:**
1. **AdGuard on ub01 (10.0.0.10):** Add DNS rewrite via API: `POST /control/rewrite/add` with `{domain, answer: "10.0.0.9"}`
2. **nginx on nuc01 (10.0.0.9):** Create server block in `/etc/nginx/sites-enabled/{domain}.conf` with `proxy_pass http://10.0.0.10:{port}`
3. **Certbot on nuc01:** `sudo certbot --nginx -d {domain}` (requires external DNS A record or CNAME pointing to public IP first)

**Note:** The AD DNS servers (10.0.0.15/.16) may already have CNAME entries for domains. Check `nslookup {domain} 10.0.0.15` before assuming setup is needed. ub01's resolv.conf uses AD DNS, not AdGuard, so `curl {domain}` from ub01 requires the AD DNS chain to resolve.

## Alembic env.py sys.path needs both local and Docker paths

**Context:** Alembic's env.py imports from `database` and `models`, which live in `backend/` locally but at `/app/` (the WORKDIR) inside Docker. The original `sys.path.insert(0, "../backend")` works locally but fails in Docker.

**Fix:** Add both paths: `sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "backend"))` AND `sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))`. The second resolves to `/app/` inside Docker.

## Nginx stale DNS after Docker container rebuild

**Context:** When an API container behind an nginx reverse proxy in Docker Compose is rebuilt (new container ID, new internal IP), the nginx container's DNS resolver cache still points to the old IP. Requests through nginx return 502 Bad Gateway even though the API is healthy on its new IP.

**Fix:** Restart the nginx/web container after rebuilding upstream containers: `docker compose restart chrysopedia-web-8096`. Alternatively, configure nginx with `resolver 127.0.0.11 valid=10s;` in the upstream block to use Docker's embedded DNS with a short TTL, but the restart is simpler for a single-admin deployment.

## Verify prior incomplete code compiles before building on it

**Context:** M005/S01/T01 found that the PipelineEvent model, migration, and instrumentation hooks had been written in a prior pass but contained critical syntax errors (missing triple-quote docstrings, unquoted string literals, reference to nonexistent `_get_session_factory()`). The code looked complete in the file listing but had never been successfully executed.

**Rule:** When a slice plan assumes prior work is complete ("model already exists, migration already applied"), verify it actually runs: import the module, run the migration, execute a basic query. Broken code that's already committed looks identical to working code in `git log`.

## Vite build-time constant injection requires JSON.stringify

**Context:** Vite's `define` config replaces identifiers at build time with string literals. If you write `define: { __APP_VERSION__: version }` where `version` is a JS string, the replacement is the raw value without quotes — the built code gets `const v = 0.1.0` (syntax error) instead of `const v = "0.1.0"`.

**Fix:** Always wrap with `JSON.stringify`: `define: { __APP_VERSION__: JSON.stringify(version) }`. Declare the constants in `vite-env.d.ts` as `declare const __APP_VERSION__: string` for TypeScript.

## Docker ARG→ENV ordering matters for Vite/Node builds

**Context:** In a Dockerfile that runs `npm run build` to produce a static frontend bundle, Vite reads environment variables at build time. A `ARG VITE_FOO=default` alone isn't visible to Node unless also set as `ENV VITE_FOO=$VITE_FOO`. The `ENV` line must appear BEFORE the `RUN npm run build` step.

**Fix:** Pattern is `ARG VITE_FOO=default` → `ENV VITE_FOO=$VITE_FOO` → `RUN npm run build`. In docker-compose.yml, pass build args: `build: args: VITE_FOO: ${HOST_VAR:-default}`.

## Slim Python Docker images lack procps (pgrep/ps)

**Context:** The `python:3.x-slim` base image (used by Dockerfile.api) does not include `procps`, so `pgrep`, `ps`, and `pidof` are unavailable. Healthcheck commands like `pgrep -f watcher.py || exit 1` silently fail because the binary doesn't exist.

**Fix:** Use Python for process healthchecks: `python -c "import os; os.kill(1, 0)" || exit 1`. This sends signal 0 to PID 1 (the container's main process), which succeeds if the process exists without actually sending a signal. Works for any single-process container where PID 1 is the service.

## PollingObserver over inotify for ZFS/NFS folder watchers

**Context:** Python watchdog's default `Observer` uses inotify on Linux, which doesn't reliably detect file changes on ZFS datasets or NFS mounts. The watcher silently misses file creation events.

**Fix:** Use `watchdog.observers.polling.PollingObserver` instead. It polls the filesystem at configurable intervals (default 1s). Slightly higher CPU usage but works reliably on any filesystem type.

## File stability detection for folder watchers receiving SCP/rsync writes

**Context:** When files land via SCP or rsync, the filesystem sees partial writes before the transfer completes. A watcher that processes on first `on_created` event may read an incomplete file.

**Fix:** After detecting a new file, check file size at two points separated by a delay (e.g., 2 seconds). Only process when the size is stable. This handles both SCP (which writes incrementally) and rsync (which may use a temp file then rename, but the rename triggers a new event with the final size).

## Check toggle state once at initialization, not per-operation

**Context:** When a feature toggle (e.g., debug mode) controls whether extra work happens on every operation (e.g., storing full LLM I/O on every API call), checking the toggle per-operation adds latency (Redis round-trip) on every call even when the feature is off.

**Fix:** Check the toggle once when creating the callback/handler closure, and let the closure capture the boolean. The trade-off is that toggle changes don't take effect until the next pipeline run (not mid-run), which is acceptable for diagnostic features.

## Resolve cross-entity links at query time, not in the frontend

**Context:** Key moment search results 404'd because the frontend navigated to `/techniques/{moment_slug}` — but moments don't have standalone pages. The frontend had no way to know the parent technique page's slug.

**Fix:** Resolve parent entity slugs at query time: use a DB JOIN for keyword search, and enrich the Qdrant payload with the parent slug during indexing for semantic search. The frontend receives a ready-to-use link target. This eliminates the need for a second API call and prevents 404 races entirely. Apply this pattern whenever a search result or list item needs to link to a parent/related entity.

## LLM-generated topic categories have inconsistent casing

**Context:** Stage 4 classification produces topic categories with inconsistent casing across different pipeline runs (e.g., 'Sound design' vs 'Sound Design'). When these are aggregated client-side (counts, grouping), they appear as separate entries.

**Fix (deferred):** Normalize casing in stage 4 output or in a post-processing step. A quick client-side fix is `.toLowerCase()` before grouping, but the root cause is upstream data quality. Worth addressing when revisiting pipeline prompt templates.

## CSS grid-template-rows 0fr/1fr for collapse/expand animation

**Context:** Animating variable-height content (accordion, expandable sections) traditionally requires JS height measurement or max-height hacks. CSS `grid-template-rows: 0fr` → `1fr` with `transition: grid-template-rows 300ms` provides smooth animation. The inner content needs `overflow: hidden; min-height: 0`.

**Fix:** Wrap the collapsible content in a grid container. Toggle between `grid-template-rows: 0fr` (collapsed) and `grid-template-rows: 1fr` (expanded). No JS measurement needed. Used in TopicsBrowse.tsx for category sections.

## Keyboard shortcut deduplication in multi-instance components

**Context:** When a component with a global keyboard shortcut (e.g., Cmd+K for search focus) is rendered in multiple places on the same page (nav + mobile panel), both instances register the shortcut, causing double-fire or unexpected focus behavior.

**Fix:** Add a `globalShortcut` boolean prop (default false). Only the primary instance sets it to true. Mobile/secondary instances render without the keyboard handler. Used in SearchAutocomplete for nav vs mobile panel instances.

## border-image strips border-radius

**Context:** CSS `border-image` property overrides `border-radius`, making rounded corners disappear. This is per CSS spec — border-image replaces the entire border rendering including corner shapes.

**Fix:** If rounded corners are required, use `box-shadow` or pseudo-elements for decorative borders instead of `border-image`. For the featured technique card, box-shadow glow provides the visual distinction even though corners are square.

## Sort strategy depends on data source homogeneity

**Context:** Search results come from two sources (Qdrant semantic + keyword SQL). Applying SQL ORDER BY is impossible when results are merged from different backends. Single-source list views (subtopic pages, creator detail) pull exclusively from PostgreSQL.

**Fix:** Use Python-level sorting (`sorted()` with key functions) for mixed-source result sets. Use SQL `ORDER BY` for single-source endpoints — it's faster and handles NULLs correctly. The sort param flows from the API query string through to whichever strategy applies. Don't try to unify into one approach.

## Multi-token AND search with partial_matches fallback

**Context:** Single-pattern ILIKE search (e.g., `%keota snare%`) only matches when the exact phrase appears in one field. Users expect "keota snare" to find content where "keota" matches the creator and "snare" matches the title/tags.

**Fix:** Tokenize by whitespace. For each token, generate OR conditions across all searchable fields (title, summary, tags, category, creator name). AND all per-token conditions. When a multi-token AND query returns zero results, fall back to partial_matches: query each token separately, score results by token coverage, return top 5 with a muted UI treatment. This gives exact matches priority while still providing useful results for imprecise queries.

## Project-root symlink + sys.path bootstrap for nested Python packages

**Context:** When a Python package lives under a subdirectory (e.g., `backend/pipeline/`), `python -m pipeline.quality` fails from the project root because `pipeline` isn't on `sys.path`. Task executors worked around this with `cd backend &&` prefix, but CI/verification gates may run from project root.

**Fix:** Create a symlink at project root (`pipeline -> backend/pipeline`) so Python finds the package. Add a `sys.path` bootstrap in the package's `__init__.py` that uses `os.path.realpath(__file__)` to resolve through the symlink and insert the real parent directory (`backend/`) onto `sys.path`. This ensures sibling imports (e.g., `from config import ...`) resolve correctly. The `realpath()` call is critical — without it, the path resolves relative to the symlink location, not the real file location.

## Offset-based citation indexing for multi-source composition

**Context:** When merging new video content into an existing technique page, both old and new key moments need citation markers ([N]) in the prose. Renumbering existing citations on every merge is error-prone and invalidates cached references.

**Fix:** Use offset-based indexing: existing moments keep [0]-[N-1], new moments get [N]-[N+M-1]. The composition prompt receives the offset explicitly. This means existing citation markers remain stable across merges — only new content gets new indices appended to the end.

## Format-discriminated rendering for evolving content schemas

**Context:** Technique pages evolved from v1 (flat dict body_sections) to v2 (list-of-objects with nesting). Migrating all existing pages at once is risky and unnecessary.

**Fix:** Add a `body_sections_format` discriminator column (default 'v1'). Frontend checks the column and selects the appropriate renderer. Both v1 and v2 renderers are independent code paths — no shared logic that could break one when editing the other. New pages get v2; existing pages stay v1 until re-processed. This pattern works for any schema evolution where old and new formats coexist.

## Compound slugs for nested anchor IDs

**Context:** When a page has both H2 sections and H3 subsections, naive slugification can produce anchor ID collisions (e.g., two different headings both slugify to "overview").

**Fix:** Use compound slugs for subsections: `sectionSlug--subSlug` (double-hyphen separator). The double-hyphen is unlikely to appear in natural headings and makes the nesting relationship visible in the URL fragment. Applied in both backend (Qdrant point IDs) and frontend (DOM element IDs).
