# S03 Research — Transcript Folder Watcher — Auto-Ingest Service

## Summary

Add a folder-watching service that monitors a directory for new transcript JSON files, validates them, and POSTs them to the existing `/api/v1/ingest` endpoint. This replaces manual curl/upload for getting transcripts into the pipeline.

The work is straightforward — filesystem monitoring + HTTP POST using well-understood Python libraries already in the project. The ingest endpoint is mature (6 integration tests, handles idempotent re-upload, auto-dispatches pipeline). The watcher just needs to be a reliable bridge from "file appears on disk" to "POST /api/v1/ingest".

## Recommendation

**watchdog** library + **httpx** (already a dependency) in a lightweight Python script, deployed as a new Docker Compose service with the watch folder bind-mounted. Polling fallback via `PollingObserver` for reliability on network-mounted filesystems.

## Implementation Landscape

### Current Ingest Flow
- `POST /api/v1/ingest` accepts multipart file upload (`UploadFile`)
- Validates JSON structure (requires `source_file`, `creator_folder`, `duration_seconds`, `segments`)
- Creates/updates Creator, upserts SourceVideo, bulk-inserts TranscriptSegments
- Saves raw JSON to `/data/transcripts/{creator_folder}/`
- Dispatches Celery pipeline via `run_pipeline.delay(str(video.id))`
- Returns `TranscriptIngestResponse` with `video_id`, `processing_status`, etc.
- Already handles idempotent re-upload (content hash matching, duplicate detection)

### Transcript JSON Format
Files produced by `whisper/transcribe.py` on hal0022:
```json
{
  "source_file": "filename.mp4",
  "creator_folder": "CreatorName",
  "duration_seconds": 2547,
  "segments": [{"start": 0.0, "end": 4.52, "text": "...", "words": [...]}]
}
```

### Deployment Topology
- All services run on **ub01** via Docker Compose (`xpltd_chrysopedia` project)
- Data volume: `/vmPool/r/services/chrysopedia_data` → mounted at `/data` in API/worker containers
- Existing transcripts live at `/vmPool/r/services/chrysopedia_data/transcripts/{CreatorName}/`
- The watcher needs a **separate watch folder** (not the transcripts store) — the ingest endpoint copies files there itself
- Proposed watch folder: `/vmPool/r/services/chrysopedia_watch/` → mounted at `/watch` in the watcher container
- API is reachable from other containers at `http://chrysopedia-api:8000` on the `chrysopedia` Docker network

### Key Design Decisions

**Watch folder ≠ transcript storage.** The ingest endpoint already saves transcripts to `/data/transcripts/`. The watch folder is an inbox — files land there, get POSTed, then get moved to a `processed/` or `failed/` subfolder. This prevents double-processing and gives visibility into what's been handled.

**File stability check.** When watchdog fires `on_created`, the file may still be being written (especially over network mounts or SCP). The watcher should wait until the file size stops changing (poll twice with a 1-2s delay) before attempting to read and POST it.

**Recursive vs flat watching.** Watch recursively — the whisper script outputs to `{output_dir}/{video_stem}.json` but users might organize into creator subfolders. The `creator_folder` field inside the JSON is authoritative, not the filesystem path.

**Error handling.** Failed files move to `failed/` with a `.error` sidecar file containing the HTTP response or exception. This avoids infinite retry loops and gives the admin a clear queue of problems to fix.

### Files to Create/Modify

| File | Action | Purpose |
|------|--------|---------|
| `backend/watcher.py` | **Create** | Main watcher script — watchdog Observer + httpx POST logic |
| `docker/Dockerfile.watcher` | **Create** | Lightweight image (could reuse Dockerfile.api or be minimal) |
| `docker-compose.yml` | **Modify** | Add `chrysopedia-watcher` service with watch folder bind mount |
| `backend/config.py` | **Modify** | Add `watch_folder_path` and `watcher_api_url` settings |

### Dependencies

- **watchdog** — needs adding to `requirements.txt` (not currently a dependency)
- **httpx** — already in `requirements.txt`, used for HTTP client
- **json** / **os** / **shutil** / **time** / **logging** — stdlib only

### Watcher Service Architecture

```
chrysopedia-watcher container:
  ┌──────────────────────────────────┐
  │  watchdog Observer               │
  │  monitors /watch/ recursively    │
  │                                  │
  │  on_created *.json:              │
  │    1. Wait for file stability    │
  │    2. Read & validate JSON keys  │
  │    3. POST to chrysopedia-api    │
  │    4. Move to processed/ or      │
  │       failed/ with sidecar       │
  └──────────────────────────────────┘
```

### Docker Compose Service

```yaml
chrysopedia-watcher:
  build:
    context: .
    dockerfile: docker/Dockerfile.api  # reuse same image
  container_name: chrysopedia-watcher
  restart: unless-stopped
  command: ["python", "watcher.py"]
  environment:
    WATCHER_API_URL: http://chrysopedia-api:8000/api/v1/ingest
    WATCH_FOLDER: /watch
  volumes:
    - /vmPool/r/services/chrysopedia_watch:/watch
  depends_on:
    chrysopedia-api:
      condition: service_healthy
  networks:
    - chrysopedia
  healthcheck:
    test: ["CMD-SHELL", "pgrep -f watcher.py || exit 1"]
    interval: 30s
    timeout: 5s
    retries: 3
```

### Natural Task Seams

1. **T01: Create watcher.py** — the core script with watchdog Observer, file stability check, JSON validation, httpx multipart POST, processed/failed file management, structured logging. This is ~80% of the work.
2. **T02: Docker integration** — Dockerfile (or reuse), compose service definition, config settings, host folder creation. Verify end-to-end: drop file → watcher detects → POSTs to API → pipeline triggers.

### Verification Strategy

- **Unit-level:** Drop a valid transcript JSON into the watch folder on ub01 → confirm watcher logs pickup, API returns 200, video appears in DB with `transcribed` status, pipeline dispatches.
- **Error case:** Drop an invalid JSON → confirm it moves to `failed/` with error sidecar, watcher continues running.
- **Stability:** Watcher survives API container restart (retry logic or health-check-gated startup).
- **Idempotency:** Drop same file twice → second copy moves to processed (API handles idempotent re-upload).

### Risks

**Low risk overall.** The ingest endpoint is battle-tested. The watcher is a thin bridge. Main concern is file stability detection on the bind mount — watchdog's inotify works well on local Linux filesystems, which is what `/vmPool/r/services/` is (ZFS on ub01). If files arrive via SCP or rsync, partial writes could trigger premature pickup — the stability check (size-stable for 2s) handles this.

### Skills

No specialized skills needed. Standard Python, Docker Compose, watchdog library. No frontend work.
