promptlooper/Auto Run Docs/01-scaffold.md

Phase 1 — Project Scaffold

Set up the PromptLooper repository, Docker infrastructure, and basic project skeleton. Read promptlooper-spec.md and CLAUDE.md before starting any task.

• Initialize the git repository at git.xpltd.co/xpltdco/promptlooper with a README.md that includes the project description from the spec, a quick-start section showing the single-container docker run command, and badges for license (AGPL-3.0) and status. Add .gitignore for Python, Node, and Docker artifacts.

  NOTE: Git repo initialized locally with remote set to git@git.xpltd.co:xpltdco/promptlooper.git. Push failed — SSH key not configured for this host or repo not yet created on Gitea. Needs manual setup before pushing.

• Create the full directory structure as defined in the spec's Project Structure section. Every directory should exist with a placeholder init.py or .gitkeep as appropriate. Include backend/, frontend/, docker/, alembic/, and all subdirectories.

  Created all directories: backend/ (with routers/, engine/adapters/, engine/scorers/, mcp/, websocket/, tests/), frontend/src/ (pages/, components/, api/), docker/, alembic/versions/. Python packages have init.py, non-Python dirs have .gitkeep.

• Create .env.example with all environment variables from the spec's Environment Variables table, with sensible defaults and comments explaining each group. Include DATABASE_URL, REDIS_URL, JWT_SECRET, DEFAULT_ENDPOINT_URL, MAX_CONCURRENT_RUNS, and all others.

  Created .env.example with all 13 environment variables organized into 7 groups (Database, Redis, Server, Auth, Default LLM Endpoint, Limits, Storage, MCP). Production-only vars (DATABASE_URL, REDIS_URL, JWT_SECRET, API_KEY, DEFAULT_ENDPOINT_*) are commented out with explanatory notes. Single-container defaults work out of the box.

• Create docker-compose.yml following XPLTD conventions: project name xpltd_promptlooper, network promptlooper (172.33.0.0/24), PostgreSQL on port 5434, Redis, API service, worker service, and web service on port 8400. Use bind mounts under /vmPool/r/services/promptlooper_* for persistent data. Model this after Chrysopedia's docker-compose.yml patterns.

  Updated existing docker-compose.yml: fixed DATABASE_URL to use standard postgresql:// scheme (not asyncpg), hardcoded DB credentials instead of requiring .env vars, added API_KEY pass-through, added working_dir for worker service, made JWT_SECRET optional with dev default. All 5 services defined: db (:5434), redis, api (MCP :8401), worker (Celery), web (:8400). Bind mounts under /vmPool/r/services/promptlooper_*. Health checks on db and redis with dependency conditions.

• Create the multi-stage Dockerfile in docker/ that builds both backend and frontend into a single image. Stage 1: Node build for frontend (npm ci && npm run build). Stage 2: Python runtime with uvicorn, copying the built frontend assets. Include nginx.conf that serves the frontend and proxies /api and /ws to uvicorn. The image should work standalone with SQLite when no DATABASE_URL is provided.

  Created 3-stage Dockerfile: (1) frontend-build with Node 20 Alpine, (2) api stage with Python 3.12-slim + uvicorn + static assets for single-container mode, (3) web stage with nginx 1.27 Alpine for production compose. nginx.conf proxies /api/ and /health to the API, upgrades /ws/ connections for WebSocket. Also created: backend/requirements.txt, frontend scaffolding (package.json, vite.config.ts, tsconfig.json, index.html, App.tsx, Tailwind config), and placeholder alembic.ini/env.py for Dockerfile COPY.

• Create backend/config.py using Pydantic Settings. Define all configuration from the Environment Variables table. Implement the SQLite fallback logic: when DATABASE_URL is not set, construct a SQLite URL pointing to DATA_DIR/promptlooper.db. When REDIS_URL is not set, set a flag for in-process mode.

  Created backend/config.py with Pydantic Settings class defining all 13 env vars. SQLite fallback via effective_database_url property constructs sqlite:///DATA_DIR/promptlooper.db when DATABASE_URL is unset. use_in_process_queue property flags in-process mode when REDIS_URL is absent. JWT_SECRET auto-generates via secrets.token_urlsafe(32) when not provided. Empty API_KEY strings normalize to None. 13 tests in tests/test_config.py all passing.

• Create backend/models.py with all SQLAlchemy ORM models from the spec's Data Model section: User, Project, Experiment, Run, StageResult, Score, ResponseCache, and WebhookConfig. Include all fields, types, relationships, and indexes. Use UUID primary keys and JSONB for flexible fields.

  Created all 8 ORM models with UUID PKs, JSON columns (using sqlalchemy.JSON for SQLite compatibility — maps to JSONB on PostgreSQL), enum types (ExperimentStatus, RunStatus), full relationship definitions with cascade deletes, and indexes on foreign keys and commonly filtered columns. Score.metadata mapped as scorer_metadata Python attribute (column name stays "metadata") to avoid SQLAlchemy reserved name conflict. 16 tests in tests/test_models.py all passing.

• Set up Alembic: create alembic.ini and alembic/env.py configured to read DATABASE_URL from the config. Generate and apply the initial migration from the models.

  Created alembic.ini with logging config and script_location pointing to alembic/. env.py reads DATABASE_URL from backend.config.settings (with override support for tests). Added script.py.mako template. Generated initial migration (e1909678e89e) with all 8 tables, indexes, foreign keys, and enums. Migration applies cleanly on SQLite (render_as_batch=True for SQLite compatibility). 5 tests in tests/test_alembic.py covering upgrade/downgrade/columns/indexes/FKs. All 34 backend tests pass.

• Create backend/schemas.py with Pydantic request/response schemas for all API endpoints. Include create/update/response schemas for Project, Experiment, Run, Endpoint, and Webhook. Include the Score input schema and export format schemas.

  Created backend/schemas.py with all Pydantic v2 schemas using ConfigDict(from_attributes=True) for ORM compatibility. Includes: Project (create/update/response/list), Experiment (create/update/response/list), Run (response/list/detail with nested stages+scores), StageResult (response), Score (input/response), Endpoint (create/update/response/list), Webhook (create/update/response/list), Auth (setup/login/token/user), Export (run row with scores dict, export response), and Health. 30 tests in tests/test_schemas.py all passing. All 64 backend tests pass.

• Create backend/main.py with the FastAPI application. Set up CORS middleware, mount all routers (even if they're stubs), configure the WebSocket endpoint, add the /health endpoint that checks DB and Redis connectivity, and add startup/shutdown lifecycle hooks.

  Created backend/main.py with: CORS middleware (allow all origins), /health endpoint checking DB (SELECT 1) and Redis (ping) connectivity, /ws WebSocket endpoint with ConnectionManager for real-time broadcasts, async lifespan hooks for DB engine + Redis init/teardown, get_db dependency yielding sessions, dynamic router mounting (silently skips missing routers). 10 tests in tests/test_main.py covering health, CORS, WebSocket connect/disconnect/echo, OpenAPI schema, 404s, broadcast, get_db, and get_redis. All 74 backend tests pass.

• Create backend/auth.py implementing JWT token generation/verification, API key validation, and the first-boot setup flow. The setup endpoint should check if any users exist — if not, accept username + password to create the admin account. Include a dependency function for route-level auth that supports both JWT and API key.

  Created backend/auth.py with: bcrypt password hashing via passlib, JWT token creation/verification (HS256, 24h expiry) using python-jose, first-boot needs_setup() + create_admin() flow (409 if admin exists), authenticate_user() for login, and get_current_user FastAPI dependency supporting both JWT Bearer tokens and X-Api-Key header (API key grants first admin user). UUID string-to-UUID conversion for SQLite compatibility. 21 tests in tests/test_auth.py covering hashing, JWT lifecycle, setup flow, login, and all auth dependency paths. All 95 backend tests pass.

• Scaffold all router files in backend/routers/ as stubs: auth.py, projects.py, experiments.py, runs.py, endpoints.py, export.py, webhooks.py, admin.py. Each should have the correct APIRouter prefix and tags, with placeholder endpoints that return 501 Not Implemented.

• Initialize the frontend: run npm create vite@latest with React + TypeScript template. Install Tailwind CSS and configure it. Install react-router-dom for routing. Create the basic App.tsx with routes for Setup, Login, Dashboard, Projects, Experiment, Live, Compare, and Admin pages (all as placeholder components). Verify it builds cleanly.

• Create frontend/src/api/client.ts with a typed API client using fetch. Include JWT token management (stored in memory, not localStorage), request/response interceptors for auth headers, and typed wrapper functions for each API endpoint group. Include WebSocket connection helper.

• Verify the full stack runs: docker compose up should start all services. The API should respond to /health. The frontend should load and show the setup screen (since no admin exists). The database migration should have run. Document any manual steps needed in the README.