---
id: S03
parent: M007
milestone: M007
provides:
  - chrysopedia-watcher Docker service running on ub01
  - Auto-ingest via file drop to /vmPool/r/services/chrysopedia_watch/
  - backend/watcher.py standalone script
requires:
  []
affects:
  []
key_files:
  - backend/watcher.py
  - backend/requirements.txt
  - docker-compose.yml
key_decisions:
  - Used httpx sync client in watcher (runs in synchronous watchdog callback threads)
  - PollingObserver over inotify Observer for ZFS/NFS reliability
  - os.kill(1,0) healthcheck instead of pgrep for slim Python images
  - SCP'd files to ub01 directly instead of git push to avoid outward-facing GitHub action
  - Ignored files in processed/ and failed/ subdirs to prevent re-processing loops
patterns_established:
  - Standalone watcher service pattern: watchdog PollingObserver + file stability check + validate + POST + disposition (processed/failed with error sidecar)
  - Reusing Dockerfile.api with command override for lightweight companion services
observability_surfaces:
  - docker logs chrysopedia-watcher — shows all watcher activity (pickup, stability wait, validation, POST result, file moves)
  - .error sidecar files in failed/ directory contain HTTP status, response body, or exception traceback
drill_down_paths:
  - .gsd/milestones/M007/slices/S03/tasks/T01-SUMMARY.md
  - .gsd/milestones/M007/slices/S03/tasks/T02-SUMMARY.md
duration: ""
verification_result: passed
completed_at: 2026-03-30T19:26:20.063Z
blocker_discovered: false
---

# S03: Transcript Folder Watcher — Auto-Ingest Service

**Built and deployed a watchdog-based folder watcher service that auto-ingests transcript JSON files dropped into a monitored directory on ub01, replacing manual curl/upload for pipeline input.**

## What Happened

This slice delivered a new Docker service (`chrysopedia-watcher`) that monitors `/vmPool/r/services/chrysopedia_watch/` on ub01 for new transcript JSON files and automatically POSTs them to the ingest API.

**T01** built `backend/watcher.py` — a standalone Python script using watchdog's `PollingObserver` (chosen over inotify for ZFS/NFS reliability). The watcher handles file stability detection (waits for size to stabilize over 2 seconds to handle partial SCP/rsync writes), validates JSON structure against required keys (`source_file`, `creator_folder`, `duration_seconds`, `segments`), POSTs valid files as multipart upload via httpx to `/api/v1/ingest`, and moves files to `processed/` or `failed/` subdirectories. Failed files get an `.error` sidecar with the failure details (HTTP status, response body, or exception traceback). Files inside `processed/` and `failed/` subdirectories are ignored to prevent re-processing loops.

**T02** wired the watcher into the Docker Compose stack as `chrysopedia-watcher`, reusing the existing `Dockerfile.api` image with a command override. The healthcheck was changed from `pgrep` (unavailable in slim Python image) to `python -c "import os; os.kill(1, 0)"`. Deployed to ub01 and verified end-to-end: valid transcript JSON auto-ingested (HTTP 200, file moved to `processed/`), invalid JSON moved to `failed/` with `.error` sidecar.

## Verification

All slice-level verification checks pass:
1. `python -m py_compile backend/watcher.py` — exits 0
2. `grep -q 'watchdog' backend/requirements.txt` — exits 0
3. `grep -q 'PollingObserver' backend/watcher.py` — exits 0
4. `grep -q 'processed' backend/watcher.py && grep -q 'failed' backend/watcher.py` — exits 0
5. `ssh ub01 "docker ps --filter name=chrysopedia-watcher --format '{{.Status}}'"` — shows "Up N minutes (healthy)"
6. End-to-end valid JSON ingestion verified on ub01 (HTTP 200, file in processed/)
7. End-to-end invalid JSON handling verified on ub01 (file in failed/ with .error sidecar)

## Requirements Advanced

- R002 — Transcript ingestion now automated via folder watcher — no manual curl/upload needed

## Requirements Validated

None.

## New Requirements Surfaced

None.

## Requirements Invalidated or Re-scoped

None.

## Deviations

Healthcheck changed from `pgrep -f watcher.py` to `python -c "import os; os.kill(1, 0)"` because procps is not available in the slim Python Docker image.

## Known Limitations

None.

## Follow-ups

None.

## Files Created/Modified

- `backend/watcher.py` — New standalone folder watcher script using watchdog PollingObserver
- `backend/requirements.txt` — Added watchdog>=4.0,<5.0 dependency
- `docker-compose.yml` — Added chrysopedia-watcher service definition
