docs(phase-7): create detailed phase plans for config consolidation (4 plans, 14 tasks)

.planning/phases/07-config-consolidation/07-01-PLAN.md

@@ -0,0 +1,395 @@
+---
+phase: 07-config-consolidation
+plan: 01
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - config/backend.yaml
+  - config/backend.yaml.example
+  - config/frontend.yaml
+  - config/frontend.yaml.example
+  - config/network.yaml
+  - config/network.yaml.example
+  - config/docker.yaml
+  - config/docker.yaml.example
+  - config/secrets.yaml.example
+  - config/README.md
+  - .gitignore
+autonomous: true
+requirements:
+  - PHASE-7-CONFIG-STRUCT
+  - PHASE-7-YAML-FORMAT
+  - PHASE-7-SECRETS-MGMT
+  - PHASE-7-DOCUMENTATION
+user_setup: []
+
+must_haves:
+  truths:
+    - "config/ folder exists with all YAML files and examples"
+    - "YAML structure matches backend, frontend, network, docker, and secrets domains"
+    - "All *.yaml.example files committed to git showing complete schema"
+    - "config/secrets.yaml is git-ignored, with example provided"
+    - ".gitignore correctly tracks examples, ignores actual secrets"
+    - "config/README.md documents every YAML file with required variables and setup instructions"
+  artifacts:
+    - path: "config/backend.yaml.example"
+      provides: "Backend configuration schema (database, AI keys, auth, logging)"
+      min_lines: 40
+    - path: "config/frontend.yaml.example"
+      provides: "Frontend configuration schema (API endpoints, feature flags, PWA)"
+      min_lines: 25
+    - path: "config/network.yaml.example"
+      provides: "Network configuration schema (ports, SSL, CORS, server IPs)"
+      min_lines: 20
+    - path: "config/docker.yaml.example"
+      provides: "Docker-specific overrides (container resources, mount paths)"
+      min_lines: 20
+    - path: "config/secrets.yaml.example"
+      provides: "Secrets template (JWT, API keys, passwords)"
+      min_lines: 15
+    - path: "config/README.md"
+      provides: "Comprehensive documentation of all config files and setup"
+      min_lines: 100
+  key_links:
+    - from: "config/backend.yaml"
+      to: "backend/config_loader.py"
+      via: "YAML parsing in config loader"
+      pattern: "load.*backend\\.yaml"
+    - from: "config/secrets.yaml.example"
+      to: ".gitignore"
+      via: "git ignore rule"
+      pattern: "config/\\*\\.yaml.*!.*\\.example"
+    - from: "config/"
+      to: "docker-compose.yml"
+      via: "volume mount in service definition"
+      pattern: "\\./config:/app/config"
+---
+
+<objective>
+Create the config/ folder structure with YAML files for backend, frontend, network, docker, and secrets configurations. Establish the foundation for centralized configuration management with clear schema documentation.
+
+Purpose: Consolidate scattered configuration (currently in inventory.env) into a structured, domain-specific YAML format (per D-01, D-02, D-03).
+
+Output: config/ folder with 4 YAML config files + examples, secrets template, comprehensive README, and .gitignore updates.
+</objective>
+
+<execution_context>
+@$HOME/.claude/get-shit-done/workflows/execute-plan.md
+@$HOME/.claude/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@.planning/phases/07-config-consolidation/07-CONTEXT.md
+@PROJECT_ARCHITECTURE.md
+@DEPLOYMENT.md
+@inventory.env
+@inventory.env.example
+@docker-compose.yml
+</context>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: Create config/ folder structure and YAML example files</name>
+  <files>
+    config/backend.yaml.example
+    config/frontend.yaml.example
+    config/network.yaml.example
+    config/docker.yaml.example
+    config/secrets.yaml.example
+  </files>
+  <read_first>
+    - inventory.env (current config source)
+    - inventory.env.example (existing schema)
+    - PROJECT_ARCHITECTURE.md (tech stack, components)
+    - DEPLOYMENT.md (current config categories)
+    - docker-compose.yml (container environment vars)
+  </read_first>
+  <action>
+    Create config/ folder in project root with 5 YAML example files (per D-01, D-03):
+
+    **config/backend.yaml.example** — Backend configuration template covering:
+    - Database: sqlite_path, log_retention_days, wal_mode (from current code patterns)
+    - AI: primary_ai_provider (gemini|claude), gemini_api_key, claude_api_key, fallback_provider
+    - Auth: jwt_secret_key, ldap_server (optional), ldap_base_dn, password_cache_path
+    - Logging: log_level (DEBUG|INFO|WARNING|ERROR), log_rotation_size_mb, log_rotation_count
+    - Application: data_dir, logs_dir, cors_origins
+    - Feature flags: ai_extraction_enabled, offline_sync_enabled, audit_logging_enabled
+
+    **config/frontend.yaml.example** — Frontend configuration template covering:
+    - API: backend_url (e.g., http://localhost:8916), timeout_ms
+    - Feature flags: service_worker_enabled, offline_enabled, ai_extraction_ui_enabled
+    - PWA: app_name, short_name, start_url, display_mode
+    - Feature toggles: enable_qr_scanner, enable_barcode_scanner, enable_batch_import
+
+    **config/network.yaml.example** — Network/deployment configuration template covering:
+    - Ports: backend_port (8916), frontend_port (8917), backend_ssl_port (8918), frontend_ssl_port (8919)
+    - SSL: ssl_enabled (true|false), certificate_path, key_path (or auto-generated by Caddy)
+    - Proxy: caddy_log_level, proxy_read_timeout_s, max_request_size_mb
+    - CORS: allowed_origins (comma-separated), allowed_methods, allowed_headers
+
+    **config/docker.yaml.example** — Docker-specific overrides covering:
+    - Images: backend_image, frontend_image, proxy_image (with tags)
+    - Resources: backend_cpu_limit, backend_memory_limit, frontend_cpu_limit, frontend_memory_limit
+    - Volumes: data_volume_driver, logs_volume_driver, use_named_volumes (true|false)
+    - Networks: network_name, network_driver (bridge|overlay)
+
+    **config/secrets.yaml.example** — Secrets template (git-ignored) covering:
+    - JWT_SECRET_KEY: "CHANGE_ME_IN_PRODUCTION" (minimum 32 chars)
+    - GEMINI_API_KEY: "your-api-key"
+    - CLAUDE_API_KEY: "your-api-key"
+    - DATABASE_PASSWORD: "db-password" (if using external DB)
+    - LDAP_PASSWORD: "ldap-password" (if using LDAP)
+    - CORS_ORIGIN_PASSWORD: (if CORS origins require auth)
+
+    All examples should have comments explaining:
+    - What each variable controls
+    - Default values
+    - Valid value ranges
+    - Where to obtain secrets (e.g., "Generate with: openssl rand -hex 32")
+    - Whether the variable is required or optional
+  </action>
+  <verify>
+    - `test -d config` (folder exists)
+    - `test -f config/backend.yaml.example && grep -q "primary_ai_provider" config/backend.yaml.example` (backend schema present)
+    - `test -f config/frontend.yaml.example && grep -q "backend_url" config/frontend.yaml.example` (frontend schema present)
+    - `test -f config/network.yaml.example && grep -q "backend_port" config/network.yaml.example` (network schema present)
+    - `test -f config/docker.yaml.example && grep -q "backend_cpu_limit" config/docker.yaml.example` (docker schema present)
+    - `test -f config/secrets.yaml.example && grep -q "JWT_SECRET_KEY" config/secrets.yaml.example` (secrets template present)
+    - All 5 files are readable and valid YAML syntax: `python3 -c "import yaml; [yaml.safe_load(open(f)) for f in ['config/backend.yaml.example', 'config/frontend.yaml.example', 'config/network.yaml.example', 'config/docker.yaml.example', 'config/secrets.yaml.example']]"`
+  </verify>
+  <done>
+    config/ folder created with 5 YAML example files, all valid YAML syntax, comprehensive comments documenting schema, required/optional status, and secret generation instructions.
+  </done>
+</task>
+
+<task type="auto">
+  <name>Task 2: Create actual config files from examples and establish .gitignore rules</name>
+  <files>
+    config/backend.yaml
+    config/frontend.yaml
+    config/network.yaml
+    config/docker.yaml
+    .gitignore
+  </files>
+  <read_first>
+    - config/backend.yaml.example (just created)
+    - config/frontend.yaml.example (just created)
+    - config/network.yaml.example (just created)
+    - config/docker.yaml.example (just created)
+    - inventory.env (current actual values to migrate)
+    - .gitignore (current ignore rules)
+  </read_first>
+  <action>
+    Create actual (non-example) YAML config files by copying examples and filling in values from inventory.env (per D-03 pattern):
+
+    **config/backend.yaml** — Copy from backend.yaml.example and fill in values from inventory.env:
+    - primary_ai_provider: (from PRIMARY_AI_PROVIDER in inventory.env, default: gemini)
+    - gemini_api_key: (from GEMINI_API_KEY if present)
+    - claude_api_key: (from CLAUDE_API_KEY if present)
+    - ldap_server: (from LDAP_SERVER if present, or empty)
+    - log_level: (from LOG_LEVEL if present, default: INFO)
+    - data_dir: ./data
+    - logs_dir: ./logs
+    - cors_origins: (from CORS_ORIGINS if present, or default: http://localhost:8917)
+
+    **config/frontend.yaml** — Create from frontend.yaml.example:
+    - backend_url: (from BACKEND_URL or derived from BACKEND_PORT, default: http://localhost:8916)
+    - timeout_ms: 30000
+    - service_worker_enabled: true
+    - offline_enabled: true
+    - ai_extraction_ui_enabled: true
+
+    **config/network.yaml** — Create from network.yaml.example:
+    - backend_port: (from BACKEND_PORT in inventory.env, default: 8916)
+    - frontend_port: (from FRONTEND_PORT in inventory.env, default: 8917)
+    - backend_ssl_port: (from BACKEND_SSL_PORT if present, default: 8918)
+    - frontend_ssl_port: (from FRONTEND_SSL_PORT if present, default: 8919)
+    - ssl_enabled: true
+    - allowed_origins: (from CORS_ORIGINS if present)
+
+    **config/docker.yaml** — Create from docker.yaml.example:
+    - backend_cpu_limit: "1.0"
+    - backend_memory_limit: "1G"
+    - frontend_cpu_limit: "0.5"
+    - frontend_memory_limit: "512M"
+    - use_named_volumes: true
+
+    **Update .gitignore** (per D-08):
+    Add rules to IGNORE actual config files (secrets protection) but TRACK examples:
+    ```
+    # Config files — ignore all .yaml except examples
+    config/*.yaml
+    !config/*.yaml.example
+    config/secrets.yaml
+    !config/secrets.yaml.example
+    ```
+
+    Do NOT delete old inventory.env yet (backward compatibility until Phase 7 complete, per D-04 deprecation timeline).
+  </action>
+  <verify>
+    - `test -f config/backend.yaml && grep -q "primary_ai_provider" config/backend.yaml` (backend.yaml exists and has content)
+    - `test -f config/frontend.yaml && grep -q "backend_url" config/frontend.yaml` (frontend.yaml exists)
+    - `test -f config/network.yaml && grep -q "backend_port" config/network.yaml` (network.yaml exists)
+    - `test -f config/docker.yaml && grep -q "backend_cpu_limit" config/docker.yaml` (docker.yaml exists)
+    - All 4 files are valid YAML: `python3 -c "import yaml; [yaml.safe_load(open(f)) for f in ['config/backend.yaml', 'config/frontend.yaml', 'config/network.yaml', 'config/docker.yaml']]"`
+    - .gitignore contains rules: `grep -q "config/\*\.yaml" .gitignore && grep -q "!config/\*\.yaml\.example" .gitignore`
+    - config/secrets.yaml does NOT exist (will be created manually by developers from example)
+  </verify>
+  <done>
+    Actual config files created with production values migrated from inventory.env, .gitignore updated to track examples and ignore actual config files (including secrets).
+  </done>
+</task>
+
+<task type="auto">
+  <name>Task 3: Create comprehensive config/README.md documentation</name>
+  <files>
+    config/README.md
+  </files>
+  <read_first>
+    - config/backend.yaml.example
+    - config/frontend.yaml.example
+    - config/network.yaml.example
+    - config/docker.yaml.example
+    - config/secrets.yaml.example
+    - DEPLOYMENT.md (current deployment docs)
+  </read_first>
+  <action>
+    Create config/README.md with comprehensive documentation (per D-08) covering:
+
+    **Section 1: Overview**
+    - Explain that config/ is the single source of truth for application configuration
+    - Mention load order: system environment variables > config/*.yaml > defaults in code
+    - Note that secrets are separate (git-ignored)
+
+    **Section 2: Quick Start**
+    - Copy all *.example files to remove .example suffix
+    - Fill in required values (especially JWT_SECRET_KEY, API keys)
+    - Create secrets.yaml from secrets.yaml.example with actual values
+
+    **Section 3: backend.yaml**
+    - Explain each variable: purpose, valid values, defaults, required/optional
+    - List how to obtain each value (e.g., "Generate JWT key with: openssl rand -hex 32")
+    - Show example values
+    - List environment variable override names (e.g., BACKEND_PRIMARY_AI_PROVIDER)
+
+    **Section 4: frontend.yaml**
+    - Explain each variable: purpose, valid values, defaults
+    - List required vs optional flags
+    - Show how feature flags affect UI behavior
+    - List environment variable override names
+
+    **Section 5: network.yaml**
+    - Explain port assignments and SSL settings
+    - Show how CORS origins work
+    - List default values and adjustment guidance
+    - List environment variable override names
+
+    **Section 6: docker.yaml**
+    - Explain container resource limits (CPU, memory)
+    - Show how to adjust for different hardware
+    - Explain volume management
+    - List environment variable override names
+
+    **Section 7: secrets.yaml**
+    - Explain git-ignore protection
+    - List all required secrets with generation/obtainment instructions
+    - Warn about security implications
+    - Show correct file permissions (600)
+
+    **Section 8: Environment Variable Override**
+    - Explain how system environment variables take precedence over YAML
+    - Show naming convention (e.g., BACKEND_PRIMARY_AI_PROVIDER -> backend.yaml:primary_ai_provider)
+    - Useful for Docker deployments where secrets come from docker run -e
+
+    **Section 9: Troubleshooting**
+    - Common issues: missing secrets, invalid YAML syntax, missing required values
+    - How to validate YAML syntax
+    - How to debug which config source is being used
+
+    Include helpful tables showing all variables, their purposes, defaults, and how to override them.
+  </action>
+  <verify>
+    - `test -f config/README.md && wc -l config/README.md | awk '{print $1}' | awk '$1 >= 100 {print "pass"}'` (README has at least 100 lines)
+    - `grep -q "Quick Start" config/README.md && grep -q "backend.yaml" config/README.md && grep -q "secrets.yaml" config/README.md` (README covers all files)
+    - `grep -q "environment variable" config/README.md` (override mechanism documented)
+    - `grep -q "JWT_SECRET_KEY" config/README.md && grep -q "openssl rand" config/README.md` (secret generation instructions present)
+  </verify>
+  <done>
+    config/README.md created with comprehensive documentation of all YAML files, required variables, generation instructions, environment variable overrides, and troubleshooting guide.
+  </done>
+</task>
+
+</tasks>
+
+<threat_model>
+## Trust Boundaries
+
+| Boundary | Description |
+|----------|-------------|
+| Filesystem → Application | Config files loaded from disk must not be tampered with |
+| Environment → Application | System environment variables override YAML (untrusted if exposed) |
+| Git repository → Deployment | .gitignore must prevent accidental commit of secrets |
+
+## STRIDE Threat Register
+
+| Threat ID | Category | Component | Disposition | Mitigation Plan |
+|-----------|----------|-----------|-------------|-----------------|
+| T-07-01 | Tampering | config/*.yaml | mitigate | File permissions enforced via git (0644 for examples, 0600 for secrets). Read-only volume mounts in Docker per docker-compose.yml line 19 `:ro` flag. |
+| T-07-02 | Information Disclosure | config/secrets.yaml | mitigate | .gitignore rule prevents accidental commit. File permissions enforced (chmod 600). config/README.md warns developers about security. Example file included to guide setup. |
+| T-07-03 | Denial of Service | config/*.yaml parsing | mitigate | PyYAML used with safe_load() only (no arbitrary code execution). Backend config_loader.py validates syntax before loading. Invalid YAML causes logged error + default fallback per D-06 load order. |
+| T-07-04 | Elevation of Privilege | JWT_SECRET_KEY exposure | accept | JWT secret hardcoded in docker-compose.yml example (line 25) warns with comment. Developers must provide production value. Risk low for development environments. |
+
+</threat_model>
+
+<verification>
+**Phase 7, Plan 1 Verification Checklist:**
+
+1. **Config Folder Structure**
+   - [ ] `config/` folder exists in project root
+   - [ ] 5 example files present: backend.yaml.example, frontend.yaml.example, network.yaml.example, docker.yaml.example, secrets.yaml.example
+   - [ ] All examples contain valid YAML syntax (parseable by python3 -m yaml)
+   - [ ] All examples have comprehensive comments explaining each variable
+
+2. **Actual Config Files**
+   - [ ] 4 actual config files exist: backend.yaml, frontend.yaml, network.yaml, docker.yaml
+   - [ ] All actual files contain valid YAML syntax
+   - [ ] Values migrated from inventory.env are present and reasonable
+   - [ ] secrets.yaml does NOT exist (will be created manually)
+
+3. **Git Integration**
+   - [ ] .gitignore updated with rules: `config/*.yaml`, `!config/*.yaml.example`, `config/secrets.yaml`, `!config/secrets.yaml.example`
+   - [ ] Examples are tracked: `git status config/*.example` shows "new file" or no changes
+   - [ ] Actual configs are ignored: `git check-ignore config/backend.yaml` returns success
+   - [ ] Secrets example is tracked but secrets themselves are ignored
+
+4. **Documentation**
+   - [ ] config/README.md exists with 100+ lines
+   - [ ] README covers all 5 YAML files (backend, frontend, network, docker, secrets)
+   - [ ] README documents environment variable override mechanism
+   - [ ] README includes secret generation/obtainment instructions
+   - [ ] README has troubleshooting section
+
+5. **Backward Compatibility (D-04)**
+   - [ ] inventory.env still exists (will be fully deprecated after backend refactor in Plan 2)
+   - [ ] No code changes yet (Config loading still uses inventory.env, Phase 7 Plan 2 updates backend)
+</verification>
+
+<success_criteria>
+- config/ folder created with 5 YAML example files defining complete schema
+- 4 actual YAML config files created with values migrated from inventory.env
+- config/secrets.yaml.example provides template (actual secrets.yaml created manually by developers)
+- .gitignore updated to track examples, ignore actual config files and secrets
+- config/README.md provides comprehensive documentation and setup instructions
+- All YAML files are syntactically valid and parseable
+- No backend code changes yet (backward compatibility maintained per D-04)
+- Foundation ready for backend refactoring in Plan 2
+</success_criteria>
+
+<output>
+After completion, create `.planning/phases/07-config-consolidation/07-01-SUMMARY.md`
+</output>