---
phase: 07-config-consolidation
plan: 02
type: execute
wave: 2
depends_on:
  - 07-01
files_modified:
  - backend/config_loader.py
  - backend/config_manager.py
  - backend/main.py
  - backend/entrypoint.sh
autonomous: true
requirements:
  - PHASE-7-BACKEND-YAML
  - PHASE-7-ENV-OVERRIDE
  - PHASE-7-NO-FALLBACK
user_setup: []

must_haves:
  truths:
    - "Backend loads from config/backend.yaml + config/secrets.yaml (YAML parsing with PyYAML)"
    - "System environment variables override YAML values (per D-06 load order)"
    - "NO fallback to inventory.env—deprecation complete after Phase 7 (per D-04)"
    - "Config loading logs which source is being used for debugging"
    - "Backend starts successfully with new config structure and passes health checks"
  artifacts:
    - path: "backend/config_loader.py"
      provides: "YAML config parsing with env var override and load order enforcement"
      exports: ["load_config()", "get_config()", "validate_config()"]
      min_lines: 80
    - path: "backend/config_manager.py"
      provides: "Config management and updates with YAML support"
      min_lines: 50
    - path: "backend/main.py"
      provides: "Updated main() to use new config_loader (no inventory.env references)"
      pattern: "from backend.config_loader import load_config"
    - path: "backend/entrypoint.sh"
      provides: "Updated Docker entrypoint sourcing YAML config paths"
      pattern: "config/backend.yaml"
  key_links:
    - from: "backend/config_loader.py"
      to: "config/backend.yaml"
      via: "PyYAML parsing"
      pattern: "yaml\\.safe_load.*backend\\.yaml"
    - from: "backend/config_loader.py"
      to: "config/secrets.yaml"
      via: "PyYAML parsing with file existence check"
      pattern: "yaml\\.safe_load.*secrets\\.yaml"
    - from: "backend/main.py"
      to: "backend/config_loader.py"
      via: "import and call load_config()"
      pattern: "from backend.config_loader import load_config"
---

<objective>
Refactor backend configuration loading from .env to YAML (backend.yaml + secrets.yaml) with system environment variable override support. Remove all inventory.env fallback paths and ensure deprecation is complete.

Purpose: Implement D-06 load order (env vars > YAML > defaults) with proper logging and validation.

Output: Updated config_loader.py with YAML parsing, config_manager.py with YAML support, main.py using new loader, and updated Docker entrypoint.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/07-config-consolidation/07-CONTEXT.md
@PROJECT_ARCHITECTURE.md
@backend/config_loader.py
@backend/config_manager.py
@backend/main.py
@backend/entrypoint.sh
@config/backend.yaml.example
@config/secrets.yaml.example
</context>

<tasks>

<task type="auto">
  <name>Task 1: Refactor backend/config_loader.py for YAML parsing with env var override</name>
  <files>
    backend/config_loader.py
  </files>
  <read_first>
    - backend/config_loader.py (current implementation using dotenv)
    - config/backend.yaml.example (schema to parse)
    - config/secrets.yaml.example (secrets schema)
    - backend/main.py (to understand how config is used)
  </read_first>
  <action>
    Refactor backend/config_loader.py to implement D-06 load order: system env vars > config/backend.yaml > config/secrets.yaml > defaults.

    **Required changes:**

    1. **Replace dotenv with PyYAML:** Add `import yaml` and remove `from dotenv import load_dotenv`

    2. **Implement load_config() function** that:
       - Locates config/ folder (one level up from backend/)
       - Attempts to load config/backend.yaml (required if exists)
       - Attempts to load config/secrets.yaml (optional, file may not exist)
       - Defines hard defaults for all variables (fallback if YAML missing)
       - Merges in order: defaults <- YAML values <- environment variable overrides
       - Returns a dict/object with all configuration

    3. **Environment variable override pattern:**
       - System env var takes precedence over YAML
       - Naming convention: BACKEND_<YAML_KEY> or just <KEY>
       - Examples:
         - BACKEND_PRIMARY_AI_PROVIDER -> backend.yaml:primary_ai_provider
         - JWT_SECRET_KEY (from secrets.yaml or env)
         - LOG_LEVEL -> backend.yaml:log_level
       - All env vars checked with os.getenv()

    4. **Load order example (pseudocode):**
       ```
       defaults = {primary_ai_provider: "gemini", log_level: "INFO", ...}
       backend_yaml = yaml.safe_load(open("config/backend.yaml")) if exists else {}
       secrets_yaml = yaml.safe_load(open("config/secrets.yaml")) if exists else {}
       config = merge(defaults, backend_yaml, secrets_yaml)
       config = merge(config, env_var_overrides())
       return config
       ```

    5. **Implement validate_config() function** that:
       - Checks required variables are present (JWT_SECRET_KEY, primary_ai_provider, etc.)
       - Validates enum values (primary_ai_provider must be gemini|claude|fallback)
       - Validates log levels (DEBUG|INFO|WARNING|ERROR)
       - Raises ConfigError if validation fails with descriptive message

    6. **Implement get_config() function** that:
       - Returns the loaded configuration dict
       - Allows other modules to import: `from backend.config_loader import get_config`

    7. **Logging:**
       - Log which files were loaded: "Loaded backend.yaml from config/"
       - Log env var overrides: "Override primary_ai_provider from environment: gemini"
       - Log final validated config (without secrets): "Config validated: primary_ai_provider=gemini, log_level=INFO"
       - Use log.info() and log.warning() (not print)

    8. **Remove all inventory.env references:**
       - Delete any checks for inventory_env_path
       - Delete fallback to backend/.env
       - NO fallback to old locations (per D-04 deprecation)

    9. **Error handling:**
       - If config/backend.yaml missing: raise ConfigError with instructions to copy from .example
       - If secrets.yaml missing: log warning but continue (secrets can come from env vars)
       - If required variables missing after all sources: raise ConfigError listing missing vars

    10. **Function signature** (updated):
        ```python
        def load_config() -> dict:
            """Load config from YAML files with env var overrides (D-06 load order)."""
        
        def get_config() -> dict:
            """Get loaded config."""
        
        def validate_config(config: dict) -> bool:
            """Validate config has all required values."""
        ```

    Keep the auto-run at module load: `load_config()` and `validate_config()` called on import.
  </action>
  <verify>
    - `grep -q "import yaml" backend/config_loader.py` (PyYAML imported)
    - `grep -q "def load_config" backend/config_loader.py && grep -q "def get_config" backend/config_loader.py` (required functions exist)
    - `grep -q "config/backend.yaml" backend/config_loader.py` (reads backend YAML)
    - `grep -q "config/secrets.yaml" backend/config_loader.py` (reads secrets YAML)
    - `grep -q "os.getenv" backend/config_loader.py` (env var overrides present)
    - `grep -v "inventory.env" backend/config_loader.py | grep -q "inventory"` should return empty (no inventory.env references)
    - `grep -q "validate_config" backend/config_loader.py` (validation function present)
    - `grep -q "log.info\|log.warning" backend/config_loader.py` (logging present)
    - File should be valid Python: `python3 -m py_compile backend/config_loader.py`
  </verify>
  <done>
    config_loader.py refactored to parse YAML files with env var override, remove inventory.env completely, implement D-06 load order with validation and logging.
  </done>
</task>

<task type="auto">
  <name>Task 2: Update backend/config_manager.py for YAML config updates</name>
  <files>
    backend/config_manager.py
  </files>
  <read_first>
    - backend/config_manager.py (current implementation)
    - config/backend.yaml.example (schema)
    - backend/config_loader.py (just updated)
  </read_first>
  <action>
    Update backend/config_manager.py to support YAML config file updates (if runtime updates are needed):

    **If config_manager.py currently reads/writes .env files:**

    1. **Replace dotenv with PyYAML:**
       - Add `import yaml`
       - Remove any dotenv usage

    2. **Implement update_config() function** that:
       - Takes key-value pairs to update
       - Loads current config/backend.yaml
       - Updates values in-memory
       - Writes back to config/backend.yaml with safe_dump()
       - NEVER writes to config/secrets.yaml (secrets are git-ignored for a reason)
       - Logs what was updated

    3. **Implement read_config() function** that:
       - Reads config/backend.yaml and returns dict
       - Uses yaml.safe_load()

    4. **Handle errors gracefully:**
       - If config/backend.yaml not found, raise error (it should exist from Plan 1)
       - If YAML syntax error, log and return current in-memory config
       - Preserve file comments if possible (or warn user they will be lost)

    5. **Function signature** (updated):
       ```python
       def read_config() -> dict:
           """Read backend.yaml and return current config."""
       
       def update_config(updates: dict) -> dict:
           """Update backend.yaml with new values and return updated config."""
       
       def validate_config_file() -> bool:
           """Validate backend.yaml syntax and required fields."""
       ```

    **If config_manager.py is minimal/unused:**
       - Add basic functions as above for future extensibility
       - Add docstrings explaining YAML handling
       - Import and use config_loader.load_config() as primary source
  </action>
  <verify>
    - `grep -q "import yaml" backend/config_manager.py` (PyYAML imported)
    - `grep -q "def read_config\|def update_config\|def validate_config_file" backend/config_manager.py` (functions present)
    - `grep -q "config/backend.yaml" backend/config_manager.py` (references YAML file)
    - `grep -q "yaml.safe_load\|yaml.safe_dump" backend/config_manager.py` (YAML parsing present)
    - File should be valid Python: `python3 -m py_compile backend/config_manager.py`
  </verify>
  <done>
    config_manager.py updated to support YAML config file updates with safe_load/safe_dump, no dotenv dependencies.
  </done>
</task>

<task type="auto">
  <name>Task 3: Update backend/main.py to use new YAML config loader</name>
  <files>
    backend/main.py
  </files>
  <read_first>
    - backend/main.py (current implementation)
    - backend/config_loader.py (just updated)
  </read_first>
  <action>
    Update backend/main.py to use the refactored config_loader:

    1. **Update imports:**
       - Replace any `from dotenv import load_dotenv` with `from backend.config_loader import load_config, get_config`
       - Remove `load_dotenv()` calls

    2. **Update main startup:**
       - Call `load_config()` at app startup (or rely on module-level auto-run)
       - Retrieve config with `get_config()` instead of `os.getenv()`
       - Example: `config = get_config()` then `db_path = config['database']['sqlite_path']`

    3. **Update environment variable access:**
       - Replace `os.getenv("BACKEND_PORT")` with `config.get("backend_port")`
       - Replace `os.getenv("JWT_SECRET_KEY")` with `config.get("jwt_secret_key")`
       - All refs should come from config dict, not os.getenv()

    4. **Remove inventory.env references:**
       - Delete any checks for inventory.env existence
       - Delete fallback logic to root-level config
       - Ensure NO hardcoded "inventory.env" strings remain

    5. **Logging:**
       - Log at startup which config was loaded (already done by config_loader, but confirm)
       - Example: "Backend initialized with config from config/backend.yaml"

    Note: This should be minimal changes if main.py already calls config_loader.load_config() at startup.
  </action>
  <verify>
    - `grep -q "from backend.config_loader import" backend/main.py` (imports from new loader)
    - `grep -q "load_dotenv" backend/main.py` should return empty (no dotenv)
    - `grep -q "inventory.env" backend/main.py` should return empty (no old config refs)
    - File should be valid Python: `python3 -m py_compile backend/main.py`
    - Check for os.getenv() calls and ensure they reference config dict instead: `grep "os.getenv" backend/main.py | head -5` (should be minimal or zero)
  </verify>
  <done>
    backend/main.py updated to import and use new YAML-based config_loader, remove all inventory.env and dotenv references.
  </done>
</task>

<task type="auto">
  <name>Task 4: Update backend/entrypoint.sh for new config paths</name>
  <files>
    backend/entrypoint.sh
  </files>
  <read_first>
    - backend/entrypoint.sh (current Docker entrypoint)
    - docker-compose.yml (volumes mapping config/)
  </read_first>
  <action>
    Update backend/entrypoint.sh to reference new config/ folder paths (per D-07 Docker support):

    1. **If entrypoint sources config:**
       - Remove any sourcing of inventory.env
       - Add comment: "Config is loaded from /app/config/ (YAML format) per Phase 7"
       - Ensure /app/config/ path is correct (mapped from host config/ via docker-compose.yml line 19)

    2. **Update environment variable documentation:**
       - Add comment: "Environment variables override YAML config (D-06 load order)"
       - List key variables that can be overridden: JWT_SECRET_KEY, PRIMARY_AI_PROVIDER, etc.
       - Example: `export JWT_SECRET_KEY="$(openssl rand -hex 32)"` (if not set)

    3. **Ensure startup doesn't fail if config missing:**
       - Add check: if config/backend.yaml not found, log error and instructions
       - Python code will raise ConfigError, so entrypoint can remain simple
       - Just ensure permissions are correct: `chmod 644 config/*.yaml`

    4. **Update Dockerfile comment (if present):**
       - Reference config/ volume mount
       - Explain YAML loading approach
  </action>
  <verify>
    - `grep -q "config/backend.yaml\|config/secrets.yaml" backend/entrypoint.sh || echo "pass"` (references new config paths or doesn't source at all)
    - `grep -q "inventory.env" backend/entrypoint.sh` should return empty (no old config)
    - File should be valid bash: `bash -n backend/entrypoint.sh`
  </verify>
  <done>
    backend/entrypoint.sh updated to reference config/ paths and document environment variable override behavior.
  </done>
</task>

</tasks>

<threat_model>
## Trust Boundaries

| Boundary | Description |
|----------|-------------|
| Filesystem → Backend | Backend reads from config/backend.yaml and config/secrets.yaml |
| Environment → Backend | System environment variables override config files (untrusted if exposed in logs) |
| Network → Backend | API receives JWT which is read from config (must be kept secret) |

## STRIDE Threat Register

| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-07-05 | Information Disclosure | Config logging | mitigate | Log config values WITHOUT secrets. config_loader.py logs final config but masks JWT_SECRET_KEY, API keys. Pattern: log keys but not values for sensitive fields. |
| T-07-06 | Denial of Service | Invalid YAML parsing | mitigate | yaml.safe_load() prevents code injection. ConfigError raised with clear message if required vars missing. validate_config() checks all required fields. |
| T-07-07 | Tampering | Environment variables | mitigate | Log env var overrides so operators know what was applied. Env vars documented in config/README.md. |
| T-07-08 | Elevation of Privilege | Backend startup | mitigate | ConfigError on missing JWT_SECRET_KEY prevents insecure defaults. Backend refuses to start without proper config. |

</threat_model>

<verification>
**Phase 7, Plan 2 Verification Checklist:**

1. **config_loader.py Refactoring**
   - [ ] PyYAML is imported, dotenv is removed
   - [ ] load_config() loads config/backend.yaml
   - [ ] load_config() loads config/secrets.yaml (optional)
   - [ ] Environment variable overrides are applied correctly (D-06)
   - [ ] validate_config() function checks required variables
   - [ ] No inventory.env references remain
   - [ ] Logging shows which config source was used
   - [ ] File is valid Python syntax

2. **config_manager.py Updates**
   - [ ] PyYAML is imported
   - [ ] read_config() and update_config() functions exist
   - [ ] YAML file operations use safe_load/safe_dump
   - [ ] No dotenv references
   - [ ] File is valid Python syntax

3. **main.py Updates**
   - [ ] Imports from backend.config_loader
   - [ ] Calls load_config() or relies on module-level auto-run
   - [ ] Uses get_config() to retrieve configuration
   - [ ] No dotenv or inventory.env references
   - [ ] No direct os.getenv() calls for app config
   - [ ] File is valid Python syntax

4. **entrypoint.sh Updates**
   - [ ] References config/ paths (not inventory.env)
   - [ ] Documents environment variable override behavior
   - [ ] Bash syntax is valid

5. **Load Order Verification (D-06)**
   - [ ] System env vars > config/backend.yaml > config/secrets.yaml > defaults
   - [ ] Integration test: set BACKEND_LOG_LEVEL=DEBUG, verify backend uses DEBUG level
   - [ ] Integration test: remove config/backend.yaml, verify defaults are used
   - [ ] Integration test: set JWT_SECRET_KEY in env, verify it overrides YAML value

6. **Deprecation Verification (D-04)**
   - [ ] inventory.env no longer used by backend
   - [ ] No fallback code remains
   - [ ] Backend fails clearly (ConfigError) if required config missing (instead of silent defaults)
</verification>

<success_criteria>
- backend/config_loader.py refactored to use PyYAML with env var override support (D-06)
- D-06 load order implemented: env vars > YAML > defaults
- All inventory.env references removed from backend code (D-04)
- config_manager.py updated for YAML file operations
- backend/main.py uses new config loader
- backend/entrypoint.sh references config/ paths
- Configuration validation ensures required variables are present
- Logging shows which sources were used for debugging
- Backend starts successfully with new config structure
</success_criteria>

<output>
After completion, create `.planning/phases/07-config-consolidation/07-02-SUMMARY.md`
</output>