--- phase: 07-config-consolidation plan: 04 type: execute wave: 3 depends_on: - 07-01 - 07-02 - 07-03 files_modified: - docker-compose.yml - backend/Dockerfile - backend/entrypoint.sh - .gitignore - DEPLOYMENT.md - README.md autonomous: true requirements: - PHASE-7-DOCKER-UPDATE - PHASE-7-DOCUMENTATION - PHASE-7-GITIGNORE user_setup: [] must_haves: truths: - "docker-compose.yml updated to reference config/ volume and remove inventory.env env_file" - "backend/Dockerfile and entrypoint updated for new config paths" - ".gitignore properly configured to track examples, ignore actual configs and secrets" - "DEPLOYMENT.md updated with YAML config structure, new Python scripts, setup instructions" - "README.md updated with config setup and configuration management instructions" - "Docker deployment works with new config structure (tested with docker-compose up)" - "All documentation references config/ as single source of truth" artifacts: - path: "docker-compose.yml" provides: "Docker Compose with config/ volume mount, no inventory.env env_file reference" pattern: "\\./config:/app/config|!inventory.env" min_lines: 120 - path: "backend/Dockerfile" provides: "Backend container image with new config paths" pattern: "config/|/app/config" - path: "backend/entrypoint.sh" provides: "Docker entrypoint with config/ reference" pattern: "config/|/app/config" - path: "DEPLOYMENT.md" provides: "Updated deployment guide with YAML config structure and Python scripts" min_lines: 150 - path: "README.md" provides: "Updated README with config setup and onboarding" min_lines: 100 - path: ".gitignore" provides: ".gitignore with rules for config/ folder (track examples, ignore secrets)" pattern: "config/.*\\.yaml" key_links: - from: "docker-compose.yml" to: "config/" via: "volume mount" pattern: "\\./config:/app/config" - from: "DEPLOYMENT.md" to: "config/README.md" provides: "Cross-reference to config documentation" pattern: "config/README.md|config/" - from: "README.md" to: "DEPLOYMENT.md" provides: "Cross-reference to deployment guide" pattern: "DEPLOYMENT.md|config/" --- Update Docker Compose, Dockerfile, documentation, and .gitignore to integrate the new config/ structure. Remove references to inventory.env from deployment infrastructure and update all deployment documentation. Purpose: Complete D-07 (Docker & Compose update), D-08 (documentation), and D-04 (deprecation) for cohesive deployment experience. Output: Updated docker-compose.yml, Dockerfile, entrypoint.sh, DEPLOYMENT.md, README.md, and .gitignore. @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md @.planning/PROJECT.md @.planning/ROADMAP.md @.planning/phases/07-config-consolidation/07-CONTEXT.md @PROJECT_ARCHITECTURE.md @docker-compose.yml @backend/Dockerfile @backend/entrypoint.sh @DEPLOYMENT.md @README.md @.gitignore @config/README.md Task 1: Update docker-compose.yml to reference config/ and remove inventory.env env_file docker-compose.yml - docker-compose.yml (current file) - backend/entrypoint.sh (to understand how config is used in containers) - config/backend.yaml.example (to understand what config is needed) - config/docker.yaml.example (Docker-specific config) Update docker-compose.yml to integrate new config/ structure (per D-07): 1. **Remove inventory.env env_file references:** - Delete or comment out `env_file: - inventory.env` from backend service (currently line 14) - Delete or comment out `env_file: - inventory.env` from frontend service (currently line 51) - Keep proxy service as is (may not need env_file) 2. **Add config/ volume mount to backend service:** - Keep existing volume mounts - Add: `- ./config:/app/config:ro` (read-only, config should not be modified in container) - Update volumes section to reflect new mount 3. **Add config/ volume mount to frontend service (if frontend needs config):** - Add: `- ./config:/app/config:ro` if frontend needs to read config files - Or skip if frontend doesn't read YAML config directly 4. **Update environment variables:** - Keep all existing environment variables (Docker overrides are still valid per D-06) - Ensure JWT_SECRET_KEY is still set with warning: `# CHANGE THIS IN PRODUCTION!` - Add comment: "Environment variables override config/backend.yaml per D-06 load order" - Ensure DATA_DIR and LOGS_DIR are set to persist volume locations 5. **Add proxy service config/ mount (if proxy reads config):** - Check if Caddyfile uses any dynamic config - If not, no change needed - If yes, add: `- ./config:/app/config:ro` 6. **Verify volume definitions:** - Named volumes (backend_data, backend_logs, frontend_logs, caddy_data, caddy_config) remain unchanged - Config mount is bind mount (./config), not named volume 7. **Add comments explaining the change:** - Add section comment before volumes: "# [D-07] New config/ structure — YAML config mounted read-only" - Add comment on env_file removal: "# [D-04] inventory.env deprecated — config now in config/ folder" - Reference Phase 7 decisions 8. **Maintain backward compatibility during transition:** - Don't delete old env_file line yet (can exist but be ignored by modern docker-compose) - Or clearly comment it out with deprecation notice Example section after update: ```yaml backend: ... # [D-04] inventory.env deprecated — see config/ folder instead # env_file: - inventory.env volumes: - backend_data:/app/data - backend_logs:/app/logs # [D-07] New config/ structure mounted read-only - ./config:/app/config:ro - ./scripts:/app/scripts:ro ... ``` - `grep -n "env_file" docker-compose.yml | head` (check if inventory.env env_file is removed/commented) - `grep -q "\\./config:/app/config:ro" docker-compose.yml` (config volume mount present) - `docker-compose config 2>&1 | grep -q "config" || echo "valid yaml"` (valid docker-compose syntax) - `grep -q "D-07\|D-04" docker-compose.yml || echo "pass"` (comments reference phase decisions, optional) docker-compose.yml updated with config/ volume mount, inventory.env env_file removed, comments documenting changes. Task 2: Update backend/Dockerfile for new config paths backend/Dockerfile - backend/Dockerfile (current file) - docker-compose.yml (just updated) - backend/entrypoint.sh (how config is used at runtime) Update backend/Dockerfile to document/support new config/ paths (per D-07): 1. **Add comments explaining config/ structure:** - Add comment at top: "# [D-07] Backend container - config/ folder mounted at /app/config (read-only)" - Add comment before WORKDIR: "# Config is expected in /app/config (mounted from host)" 2. **Ensure volume mount points exist:** - Config is mounted at runtime by docker-compose, not created in Dockerfile - No changes needed to RUN commands for config directory - (Already handled by docker-compose volume mount) 3. **Update any hardcoded paths referencing inventory.env:** - Search for "inventory.env" in Dockerfile - Replace with reference to config/ or remove if no longer needed - Example: If old CMD references inventory.env, update to reference config/ 4. **Update ENTRYPOINT or CMD if needed:** - Ensure entrypoint.sh (or equivalent) references config/ paths - Add environment documentation: "# Config sources: /app/config/backend.yaml, /app/config/secrets.yaml, environment variables" 5. **Add healthcheck if not present:** - Verify backend has healthcheck (curl to /health endpoint) - Should already be in docker-compose.yml, but double-check Dockerfile 6. **Document environment variables:** - Add comment: "# Environment variables override YAML config per D-06" - List: DATA_DIR, LOGS_DIR, LOG_LEVEL (these come from env and/or config) Example after update: ```dockerfile # [D-07] Backend container - config/ folder mounted at /app/config (read-only) # Config sources: /app/config/backend.yaml, /app/config/secrets.yaml, environment variables # Environment variables override YAML config per D-06 load order FROM python:3.12-slim WORKDIR /app # Copy code, requirements, and startup scripts COPY backend/ ./backend/ COPY scripts/ ./scripts/ COPY requirements.txt . # Install dependencies RUN pip install --no-cache-dir -r requirements.txt # Config is mounted at /app/config by docker-compose # No need to COPY config/ here (it's mounted read-only) ENTRYPOINT ["python", "backend/main.py"] ``` - `grep -q "D-07\|config/" backend/Dockerfile || echo "pass"` (comments reference config, optional) - `grep -q "inventory.env" backend/Dockerfile` should return empty (no old inventory.env refs) - `docker build -f backend/Dockerfile .` (valid Dockerfile syntax — may not succeed without full context, but no syntax errors) backend/Dockerfile updated with comments documenting config/ structure, no inventory.env references. Task 3: Update backend/entrypoint.sh for new config paths backend/entrypoint.sh - backend/entrypoint.sh (current file) - backend/config_loader.py (updated in Plan 2, to understand config loading) - config/README.md (documentation on config structure) Update backend/entrypoint.sh to reference and support new config/ structure (per D-07): 1. **Remove inventory.env sourcing:** - Delete any lines that source or check for inventory.env - Delete any EXPORT statements that copy inventory.env values to environment 2. **Add config/ path documentation:** - Add comment at top: "# [D-07] Backend entrypoint - loads config from /app/config/ (YAML format)" - Add comment: "# Config sources: /app/config/backend.yaml, /app/config/secrets.yaml, environment variables" 3. **Add environment variable override documentation:** - Add comment: "# [D-06] Environment variables override YAML config — set below takes precedence" - List typical overrides: JWT_SECRET_KEY, PRIMARY_AI_PROVIDER, LOG_LEVEL, etc. 4. **Ensure config validation:** - Add check: if [ ! -f "/app/config/backend.yaml" ]; then log error and instructions - Add comment: "# Config validation handled by Python config_loader.py" 5. **Set working directory:** - Ensure WORKDIR is set to /app (should be done in Dockerfile, but double-check) 6. **Exec main process:** - Ensure entrypoint uses `exec` to replace shell: `exec python backend/main.py` - This ensures signals (SIGTERM) are properly handled by Python process Example after update: ```bash #!/bin/bash # [D-07] Backend entrypoint - loads config from /app/config/ (YAML format) # Config sources: /app/config/backend.yaml, /app/config/secrets.yaml, environment variables # [D-06] Environment variables override YAML config (below takes precedence) set -euo pipefail cd /app # Verify config is accessible if [ ! -f "/app/config/backend.yaml" ]; then echo "[ERROR] /app/config/backend.yaml not found!" echo "[ERROR] Config must be mounted from host at /app/config/" echo "[ERROR] See config/README.md for setup instructions" exit 1 fi # Environment variables below override YAML config # (docker run -e JWT_SECRET_KEY="..." or docker-compose environment) # Start backend (signals properly handled with exec) exec python -m uvicorn backend.main:app --host 0.0.0.0 --port 8000 ``` 7. **Keep it minimal:** - Entrypoint should be simple (most logic in Python config_loader.py) - Just verify config exists and start the app - `grep -q "D-07\|D-06\|config/" backend/entrypoint.sh` (references config structure, optional) - `grep -q "inventory.env" backend/entrypoint.sh` should return empty (no old config) - `bash -n backend/entrypoint.sh` (valid bash syntax) - `head -1 backend/entrypoint.sh | grep -q "bash"` (shebang present) backend/entrypoint.sh updated to reference config/ paths, remove inventory.env, document env var overrides. Task 4: Update .gitignore to track config examples and ignore actual configs/secrets .gitignore - .gitignore (current file) - config/backend.yaml.example (created in Plan 1) - config/secrets.yaml.example (created in Plan 1) Update .gitignore to properly handle config/ folder (per D-03, D-08): 1. **Add config/ rules:** ``` # [D-08] Config folder — track examples, ignore actual configs and secrets config/*.yaml !config/*.yaml.example config/secrets.yaml !config/secrets.yaml.example ``` 2. **Rationale:** - `config/*.yaml` — Ignore all YAML files (actual configs with real secrets) - `!config/*.yaml.example` — Except examples (these are tracked for schema/documentation) - `config/secrets.yaml` — Explicitly ignore secrets file (redundant but clear) - `!config/secrets.yaml.example` — Except example (for developer setup guidance) 3. **Clean up old rules:** - Remove any existing `inventory.env` entries from .gitignore (deprecated) - Or update to comment them as deprecated: `# inventory.env # [D-04] Deprecated - use config/backend.yaml` 4. **Add comment at top of config section:** - Add comment: "# [D-04] inventory.env deprecated — see config/ folder instead" - Add comment: "# [D-08] Config structure: examples tracked, actual configs ignored" 5. **Example .gitignore config section:** ``` # [D-04] inventory.env deprecated — see config/ folder instead # [D-08] Config structure: examples tracked (schema), actual configs ignored (secrets) config/*.yaml !config/*.yaml.example config/secrets.yaml !config/secrets.yaml.example ``` 6. **Verify git status:** - After update, git status should show: - config/*.yaml.example as "new file" or tracked - config/*.yaml as ignored - config/secrets.yaml as ignored - `grep -q "config/\\*\\.yaml" .gitignore` (config rule present) - `grep -q "!config/\\*\\.yaml\\.example" .gitignore` (exception for examples present) - `grep -q "config/secrets\\.yaml" .gitignore` (secrets ignored) - `grep -q "!config/secrets\\.yaml\\.example" .gitignore` (example tracked) .gitignore updated with config/ rules to track examples and ignore actual configs/secrets. Task 5: Update DEPLOYMENT.md with YAML config structure and new Python scripts DEPLOYMENT.md - DEPLOYMENT.md (current file) - config/README.md (created in Plan 1) - scripts/deploy.py, scripts/run_standalone.py (created in Plan 3) Update DEPLOYMENT.md to document new YAML config structure and Python deployment scripts (per D-08): 1. **Add section: Configuration (Before Quick Start)** - Explain config/ as single source of truth - List config files: backend.yaml, frontend.yaml, network.yaml, docker.yaml, secrets.yaml - Reference config/README.md for detailed setup - Explain examples and how to create actual config from examples 2. **Update Quick Start section:** - Step 1: Clone and cd - Step 2: Copy config examples to actual files: `cp config/*.yaml.example {without .example}` - Step 3: Edit config files (backend.yaml, network.yaml, secrets.yaml) with your values - Step 4: Choose deployment mode (Docker or Standalone) 3. **Update Docker Deployment section:** - Replace old deploy.sh instructions with new deploy.py - Usage: `python3 scripts/deploy.py [production|staging|development] [--rebuild]` - Script handles: pre-flight checks, port validation, config loading, health checks - Output: service status, access URLs 4. **Update Standalone Deployment section:** - Replace old run_standalone.sh with new run_standalone.py - Usage: `python3 scripts/run_standalone.py [--backend-only|--frontend-only]` - Script handles: config loading, backend startup, frontend startup, signal handling 5. **Update Configuration Reference section:** - Refer to config/README.md for complete reference - List key files: config/backend.yaml, config/frontend.yaml, config/network.yaml, config/docker.yaml, config/secrets.yaml - Explain environment variable overrides (D-06) 6. **Add Systemd Service Installation section:** - Usage: `sudo python3 scripts/install_service.py [--user=www-data]` - Explain what service does: runs standalone backend + frontend - Show how to manage: `systemctl start|stop|status ainventory` - Show logs: `journalctl -u ainventory -f` 7. **Add Backup & Export section:** - Usage: `python3 scripts/export_prod.py [--output=/path/to/backup.tar.gz] [--include-logs]` - Explains what is included: data, config, config templates - Explains what is excluded: actual secrets (security), logs (optional) 8. **Add Security section:** - Mention secrets.yaml is git-ignored - Explain how to set up secrets (copy from example, fill in values) - Warn about JWT_SECRET_KEY in docker-compose.yml (must change for production) 9. **Add Troubleshooting section:** - Common issues: missing config files, invalid YAML syntax, port conflicts - Debug steps: check config syntax, verify file permissions, run health checks - Reference config/README.md for setup help 10. **Add Migration section (from old inventory.env):** - Explain Phase 7 transition from inventory.env to config/ structure - Provide migration script or manual steps - Clear instructions: which old values map to which config files Structure should be roughly: - 1. Overview (unchanged) - 2. Prerequisites (unchanged) - **3. Configuration** (NEW) - 4. Quick Start (UPDATED) - 5. Deployment Modes (Docker, Standalone) (UPDATED to use Python scripts) - 6. Systemd Service (UPDATED with Python script) - 7. Backup & Export (UPDATED with Python script) - 8. Operations & Health Monitoring (UPDATED with new paths) - 9. Security (NEW or UPDATED) - 10. Troubleshooting (UPDATED) - 11. Migration from inventory.env (NEW) - `grep -q "config/.*\\.yaml" DEPLOYMENT.md` (references YAML config files) - `grep -q "scripts/deploy\\.py\|scripts/run_standalone\\.py" DEPLOYMENT.md` (references Python scripts) - `grep -q "config/README\\.md" DEPLOYMENT.md` (cross-references config documentation) - `grep -q "environment.*override\|D-06" DEPLOYMENT.md || echo "pass"` (explains env var overrides, optional) - File should have 150+ lines: `wc -l DEPLOYMENT.md | awk '$1 >= 150 {print "pass"}'` DEPLOYMENT.md updated with YAML config structure, Python script usage, systemd service, backup procedures, and troubleshooting. Task 6: Update README.md with config setup and configuration management instructions README.md - README.md (current file) - DEPLOYMENT.md (just updated) - config/README.md (created in Plan 1) Update README.md to include configuration management and quick setup instructions (per D-08): 1. **Add Configuration section (after Quick Start or before Deployment):** - Brief explanation: config/ folder is single source of truth - Quick steps: cp config/*.example to remove .example, edit with your values - Reference config/README.md for detailed setup - Reference DEPLOYMENT.md for deployment options 2. **Update Quick Start section (if exists):** - Add configuration step before deployment - Example: ``` # 1. Clone and setup git clone ... && cd tfm-inventory # 2. Configure application cp config/*.yaml.example config/$(basename {} .example) # or similar nano config/backend.yaml # Edit with your values cp config/secrets.yaml.example config/secrets.yaml nano config/secrets.yaml # Fill in API keys and secrets # 3. Deploy (choose one) python3 scripts/deploy.py production # Docker # OR python3 scripts/run_standalone.py # Standalone ``` 3. **Add note about .gitignore:** - Config examples are tracked (for schema) - Actual configs are git-ignored (protect secrets) - secrets.yaml is git-ignored (never commit) 4. **Cross-reference documentation:** - Add links/references to: - config/README.md (configuration reference) - DEPLOYMENT.md (detailed deployment guide) - dev_docs/ (for development setup) 5. **Add "Getting Help" section (if not exists):** - Point to DEPLOYMENT.md troubleshooting - Point to config/README.md for config questions - Reference AI_RULES.md for project conventions 6. **Update any hardcoded inventory.env references:** - Replace with config/ references - Update any env-related docs/examples 7. **Maintain existing structure:** - Don't delete or significantly reorder existing sections - Just add/update config-related content and update cross-references Keep README concise but informative. Detailed docs go in DEPLOYMENT.md and config/README.md. - `grep -q "config/" README.md` (references config structure) - `grep -q "DEPLOYMENT\\.md\|config/README\\.md" README.md` (cross-references detailed docs) - `grep -q "secrets.yaml\|git.*ignore" README.md || echo "pass"` (mentions secrets and gitignore, optional) - File should be valid markdown: `grep "^#" README.md | head -3` (has headers) README.md updated with configuration management, quick setup steps, and documentation cross-references. ## Trust Boundaries | Boundary | Description | |----------|-------------| | Git repository → Deployment | .gitignore must prevent secrets from being committed | | Documentation → Users | Documentation must clearly explain security requirements | | Environment → Container | Docker environment variables can expose secrets if logged | ## STRIDE Threat Register | Threat ID | Category | Component | Disposition | Mitigation Plan | |-----------|----------|-----------|-------------|-----------------| | T-07-13 | Tampering | docker-compose volume mount | mitigate | Config volume mounted read-only `:ro`. Backend cannot modify config at runtime. Changes require host-level edits. | | T-07-14 | Information Disclosure | DEPLOYMENT.md instructions | mitigate | Documentation warns to generate JWT_SECRET_KEY, don't use placeholder. Warns about secrets.yaml setup. | | T-07-15 | Information Disclosure | .gitignore config rules | mitigate | Clear rules prevent accidental secret commits. !config/*.example exception ensures schema is tracked. | | T-07-16 | Elevation of Privilege | Docker container permissions | mitigate | No RUN as root in Dockerfile. Container runs as unprivileged user (if specified in docker-compose). | **Phase 7, Plan 4 Verification Checklist:** 1. **docker-compose.yml** - [ ] inventory.env env_file removed or commented (per D-04) - [ ] ./config:/app/config:ro volume mount added to backend service - [ ] Syntax valid: `docker-compose config` succeeds - [ ] Comments reference D-07, D-04 decisions - [ ] Environment variables preserved (JWT_SECRET_KEY etc. with production warning) 2. **backend/Dockerfile** - [ ] No references to inventory.env - [ ] Comments reference config/ structure and D-07 - [ ] ENTRYPOINT or CMD properly set - [ ] Syntax valid: `docker build --dry-run` or manual parse 3. **backend/entrypoint.sh** - [ ] No sourcing of inventory.env - [ ] References /app/config/ paths - [ ] Checks if config/backend.yaml exists - [ ] Documents environment variable override behavior - [ ] Bash syntax valid: `bash -n backend/entrypoint.sh` 4. **.gitignore** - [ ] Rules added: config/*.yaml, !config/*.yaml.example, config/secrets.yaml, !config/secrets.yaml.example - [ ] Old inventory.env references removed or marked deprecated - [ ] Git test: `git check-ignore config/backend.yaml` returns success (ignored) - [ ] Git test: `git status config/*.example` shows untracked (not ignored) 5. **DEPLOYMENT.md** - [ ] Configuration section added before or after Quick Start - [ ] References config/ files (backend.yaml, frontend.yaml, network.yaml, docker.yaml, secrets.yaml) - [ ] Docker deployment updated to use scripts/deploy.py - [ ] Standalone deployment updated to use scripts/run_standalone.py - [ ] Systemd service section with scripts/install_service.py - [ ] Backup section with scripts/export_prod.py - [ ] Troubleshooting section - [ ] Migration section (from inventory.env to config/) - [ ] Length 150+ lines 6. **README.md** - [ ] Configuration section added - [ ] Quick start updated with config setup steps - [ ] Cross-references to config/README.md and DEPLOYMENT.md - [ ] Mentions secrets.yaml and .gitignore - [ ] No hardcoded inventory.env references 7. **Cross-document consistency** - [ ] README.md, DEPLOYMENT.md, config/README.md use consistent terminology - [ ] All three documents reference each other appropriately - [ ] No conflicting instructions across documents - docker-compose.yml updated with config/ volume mount, inventory.env env_file removed (D-07) - backend/Dockerfile and entrypoint.sh updated for new config paths - .gitignore configured to track examples, ignore actual configs and secrets (D-08) - DEPLOYMENT.md updated with YAML structure, Python scripts, systemd setup, troubleshooting (D-08) - README.md updated with config setup and documentation cross-references (D-08) - All documentation references config/ as single source of truth - Docker deployment works with new config structure - Clear migration path from inventory.env to new config/ structure documented After completion, create `.planning/phases/07-config-consolidation/07-04-SUMMARY.md`