# S03: Transcript Folder Watcher — Auto-Ingest Service

**Goal:** A folder-watching service monitors a directory for new transcript JSON files, validates them, and POSTs them to the /api/v1/ingest endpoint, replacing manual curl/upload for getting transcripts into the pipeline.
**Demo:** After this: Drop a transcript JSON into the watch folder on ub01 → file is detected, validated, POSTed to /ingest, pipeline triggers automatically

## Tasks
- [x] **T01: Built backend/watcher.py with PollingObserver-based folder watching, file stability detection, JSON validation, multipart POST to /api/v1/ingest, and processed/failed file disposition with error sidecars** — Create the core watcher script that monitors a folder for new transcript JSON files and POSTs them to the ingest API.

**Slice:** S03 — Transcript Folder Watcher — Auto-Ingest Service
**Milestone:** M007

## Description

Build `backend/watcher.py` — a standalone Python script using the `watchdog` library to monitor a configurable folder for new `.json` files. When a file is detected:
1. Wait for file stability (size unchanged for 2 seconds) to handle SCP/rsync partial writes
2. Read and validate JSON structure (must contain `source_file`, `creator_folder`, `duration_seconds`, `segments`)
3. POST the file as multipart upload to the ingest API endpoint using httpx
4. Move successfully processed files to `processed/` subfolder
5. Move failed files to `failed/` subfolder with a `.error` sidecar file containing the error details

The script uses `PollingObserver` (not inotify Observer) for reliability on ZFS/network-mounted filesystems. It watches recursively.

Add `watchdog` to `backend/requirements.txt`.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| Ingest API | Move file to `failed/` with HTTP status + body in `.error` sidecar | httpx timeout (30s default) → same as error | Log warning, move to `failed/` |
| Filesystem | Log exception, skip file (watchdog continues) | N/A | N/A |

## Negative Tests

- **Malformed inputs**: Invalid JSON (parse error) → `failed/` with error sidecar. Missing required keys → `failed/` with validation error. Non-JSON files → ignored (only `.json` suffix triggers processing). Empty file → `failed/`.
- **Error paths**: API returns 4xx/5xx → `failed/` with response body. API unreachable → `failed/` with connection error. File deleted between detection and read → log warning, skip.
- **Boundary conditions**: File still being written (size changing) → wait until stable. Same filename in `processed/` already exists → overwrite (idempotent).

## Steps

1. Add `watchdog>=4.0,<5.0` to `backend/requirements.txt`
2. Create `backend/watcher.py` with:
   - Configuration from environment variables: `WATCH_FOLDER` (default `/watch`), `WATCHER_API_URL` (default `http://chrysopedia-api:8000/api/v1/ingest`), `WATCHER_STABILITY_SECONDS` (default `2`), `WATCHER_POLL_INTERVAL` (default `5`)
   - `TranscriptHandler(FileSystemEventHandler)` class that handles `on_created` events for `.json` files
   - `wait_for_stability(path, seconds)` function — polls file size every 0.5s, returns True when size is unchanged for `seconds` duration, returns False if file disappears
   - `validate_transcript(data)` function — checks for required keys (`source_file`, `creator_folder`, `duration_seconds`, `segments`)
   - `post_to_ingest(filepath)` function — reads file, validates JSON, POSTs as multipart upload via httpx, returns (success, detail)
   - `move_to_processed(filepath)` / `move_to_failed(filepath, error)` functions — create subdirs as needed, move file, write `.error` sidecar for failures
   - Main block: create `processed/` and `failed/` subdirs, start `PollingObserver` with recursive=True, run until KeyboardInterrupt
   - Structured logging with `logging.basicConfig` at INFO level, logger name `watcher`
3. Verify `python -m py_compile backend/watcher.py` passes

## Must-Haves

- [x] watchdog added to requirements.txt
- [x] watcher.py handles file stability check
- [x] watcher.py validates JSON keys before POST
- [x] watcher.py POSTs via httpx multipart upload
- [x] Processed files move to processed/ subfolder
- [x] Failed files move to failed/ with .error sidecar
- [x] Structured logging with watcher.* logger
- [x] PollingObserver used (not inotify) for ZFS reliability

## Verification

- `python -m py_compile backend/watcher.py` exits 0
- `grep -q 'watchdog' backend/requirements.txt` exits 0
- `grep -q 'PollingObserver' backend/watcher.py` exits 0
- `grep -q 'processed' backend/watcher.py && grep -q 'failed' backend/watcher.py` exits 0

## Observability Impact

- Signals added: `watcher` logger with structured messages for pickup, stability wait, validation, POST result, and file moves
- How a future agent inspects this: `docker logs chrysopedia-watcher` shows all watcher activity
- Failure state exposed: `.error` sidecar files in `failed/` directory with HTTP status, response body, or exception traceback
  - Estimate: 45m
  - Files: backend/watcher.py, backend/requirements.txt
  - Verify: python -m py_compile backend/watcher.py && grep -q 'watchdog' backend/requirements.txt && grep -q 'PollingObserver' backend/watcher.py
- [x] **T02: Added chrysopedia-watcher service to Docker Compose, deployed and verified end-to-end file-drop auto-ingest on ub01** — Add the chrysopedia-watcher service to docker-compose.yml, create the watch folder on ub01, build and deploy, and verify end-to-end file-drop ingestion.

**Slice:** S03 — Transcript Folder Watcher — Auto-Ingest Service
**Milestone:** M007

## Description

Wire the watcher script into the Docker Compose stack as a new service. The watcher reuses the existing `Dockerfile.api` image (same Python + deps) but runs `python watcher.py` instead of uvicorn. The watch folder on ub01 is `/vmPool/r/services/chrysopedia_watch/` bind-mounted to `/watch` in the container.

This task deploys to ub01 and verifies end-to-end: drop a transcript JSON → watcher picks it up → POSTs to API → video record created.

**Important context for executor:** The canonical codebase lives on ub01 at `/vmPool/r/repos/xpltdco/chrysopedia`. You must SSH to ub01 to edit docker-compose.yml, build, and test. The watcher.py from T01 needs to be committed and available in the repo on ub01.

## Failure Modes

| Dependency | On error | On timeout | On malformed response |
|------------|----------|-----------|----------------------|
| Docker build | Fix Dockerfile or requirements, rebuild | N/A | N/A |
| API healthcheck | Watcher waits (depends_on condition: service_healthy) | Start timeout → container restarts | N/A |

## Steps

1. SSH to ub01 and add the `chrysopedia-watcher` service to `docker-compose.yml`:
   - `build: context: . dockerfile: docker/Dockerfile.api` (reuse existing image)
   - `container_name: chrysopedia-watcher`
   - `restart: unless-stopped`
   - `command: ["python", "watcher.py"]`
   - `environment: WATCHER_API_URL: http://chrysopedia-api:8000/api/v1/ingest, WATCH_FOLDER: /watch`
   - `volumes: /vmPool/r/services/chrysopedia_watch:/watch`
   - `depends_on: chrysopedia-api: condition: service_healthy`
   - `networks: chrysopedia`
   - `healthcheck: test: ["CMD-SHELL", "pgrep -f watcher.py || exit 1"] interval: 30s timeout: 5s retries: 3`
2. Create the watch folder on ub01: `mkdir -p /vmPool/r/services/chrysopedia_watch`
3. Ensure T01's `backend/watcher.py` and updated `backend/requirements.txt` are committed and pushed to the repo on ub01 (git pull if needed)
4. Build and deploy: `cd /vmPool/r/repos/xpltdco/chrysopedia && docker compose build chrysopedia-watcher && docker compose up -d chrysopedia-watcher`
5. Verify watcher is running: `docker exec chrysopedia-watcher pgrep -f watcher.py`
6. End-to-end test: Copy a known-good transcript JSON to `/vmPool/r/services/chrysopedia_watch/test_ingest.json` → check watcher logs for pickup and 200 → verify file moved to `processed/`
7. Error test: Create an invalid JSON file in the watch folder → verify it moves to `failed/` with `.error` sidecar

## Must-Haves

- [x] chrysopedia-watcher service defined in docker-compose.yml
- [x] Watch folder bind mount configured
- [x] Watcher container starts and stays healthy
- [x] End-to-end file-drop ingestion verified
- [x] Failed file handling verified

## Verification

- `ssh ub01 "docker ps --filter name=chrysopedia-watcher --format '{{.Status}}'"` shows healthy/running
- `ssh ub01 "docker logs chrysopedia-watcher 2>&1 | tail -5"` shows watcher startup message
- `ssh ub01 "ls /vmPool/r/services/chrysopedia_watch/processed/"` contains the test file
- `ssh ub01 "ls /vmPool/r/services/chrysopedia_watch/failed/"` contains the invalid test file + `.error` sidecar

## Inputs

- `backend/watcher.py` — watcher script from T01
- `backend/requirements.txt` — updated with watchdog dependency from T01
- `docker-compose.yml` — existing compose file to modify

## Expected Output

- `docker-compose.yml` — updated with chrysopedia-watcher service definition
  - Estimate: 30m
  - Files: docker-compose.yml
  - Verify: ssh ub01 "docker ps --filter name=chrysopedia-watcher --format '{{.Status}}'" | grep -qi 'healthy\|up'
