18 KiB
18 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, user_setup, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | user_setup | must_haves | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 07-config-consolidation | 02 | execute | 2 |
|
|
true |
|
|
Purpose: Implement D-06 load order (env vars > YAML > defaults) with proper logging and validation.
Output: Updated config_loader.py with YAML parsing, config_manager.py with YAML support, main.py using new loader, and updated Docker entrypoint.
<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/phases/07-config-consolidation/07-CONTEXT.md @PROJECT_ARCHITECTURE.md @backend/config_loader.py @backend/config_manager.py @backend/main.py @backend/entrypoint.sh @config/backend.yaml.example @config/secrets.yaml.example Task 1: Refactor backend/config_loader.py for YAML parsing with env var override backend/config_loader.py - backend/config_loader.py (current implementation using dotenv) - config/backend.yaml.example (schema to parse) - config/secrets.yaml.example (secrets schema) - backend/main.py (to understand how config is used) Refactor backend/config_loader.py to implement D-06 load order: system env vars > config/backend.yaml > config/secrets.yaml > defaults.**Required changes:**
1. **Replace dotenv with PyYAML:** Add `import yaml` and remove `from dotenv import load_dotenv`
2. **Implement load_config() function** that:
- Locates config/ folder (one level up from backend/)
- Attempts to load config/backend.yaml (required if exists)
- Attempts to load config/secrets.yaml (optional, file may not exist)
- Defines hard defaults for all variables (fallback if YAML missing)
- Merges in order: defaults <- YAML values <- environment variable overrides
- Returns a dict/object with all configuration
3. **Environment variable override pattern:**
- System env var takes precedence over YAML
- Naming convention: BACKEND_<YAML_KEY> or just <KEY>
- Examples:
- BACKEND_PRIMARY_AI_PROVIDER -> backend.yaml:primary_ai_provider
- JWT_SECRET_KEY (from secrets.yaml or env)
- LOG_LEVEL -> backend.yaml:log_level
- All env vars checked with os.getenv()
4. **Load order example (pseudocode):**
```
defaults = {primary_ai_provider: "gemini", log_level: "INFO", ...}
backend_yaml = yaml.safe_load(open("config/backend.yaml")) if exists else {}
secrets_yaml = yaml.safe_load(open("config/secrets.yaml")) if exists else {}
config = merge(defaults, backend_yaml, secrets_yaml)
config = merge(config, env_var_overrides())
return config
```
5. **Implement validate_config() function** that:
- Checks required variables are present (JWT_SECRET_KEY, primary_ai_provider, etc.)
- Validates enum values (primary_ai_provider must be gemini|claude|fallback)
- Validates log levels (DEBUG|INFO|WARNING|ERROR)
- Raises ConfigError if validation fails with descriptive message
6. **Implement get_config() function** that:
- Returns the loaded configuration dict
- Allows other modules to import: `from backend.config_loader import get_config`
7. **Logging:**
- Log which files were loaded: "Loaded backend.yaml from config/"
- Log env var overrides: "Override primary_ai_provider from environment: gemini"
- Log final validated config (without secrets): "Config validated: primary_ai_provider=gemini, log_level=INFO"
- Use log.info() and log.warning() (not print)
8. **Remove all inventory.env references:**
- Delete any checks for inventory_env_path
- Delete fallback to backend/.env
- NO fallback to old locations (per D-04 deprecation)
9. **Error handling:**
- If config/backend.yaml missing: raise ConfigError with instructions to copy from .example
- If secrets.yaml missing: log warning but continue (secrets can come from env vars)
- If required variables missing after all sources: raise ConfigError listing missing vars
10. **Function signature** (updated):
```python
def load_config() -> dict:
"""Load config from YAML files with env var overrides (D-06 load order)."""
def get_config() -> dict:
"""Get loaded config."""
def validate_config(config: dict) -> bool:
"""Validate config has all required values."""
```
Keep the auto-run at module load: `load_config()` and `validate_config()` called on import.
**If config_manager.py currently reads/writes .env files:**
1. **Replace dotenv with PyYAML:**
- Add `import yaml`
- Remove any dotenv usage
2. **Implement update_config() function** that:
- Takes key-value pairs to update
- Loads current config/backend.yaml
- Updates values in-memory
- Writes back to config/backend.yaml with safe_dump()
- NEVER writes to config/secrets.yaml (secrets are git-ignored for a reason)
- Logs what was updated
3. **Implement read_config() function** that:
- Reads config/backend.yaml and returns dict
- Uses yaml.safe_load()
4. **Handle errors gracefully:**
- If config/backend.yaml not found, raise error (it should exist from Plan 1)
- If YAML syntax error, log and return current in-memory config
- Preserve file comments if possible (or warn user they will be lost)
5. **Function signature** (updated):
```python
def read_config() -> dict:
"""Read backend.yaml and return current config."""
def update_config(updates: dict) -> dict:
"""Update backend.yaml with new values and return updated config."""
def validate_config_file() -> bool:
"""Validate backend.yaml syntax and required fields."""
```
**If config_manager.py is minimal/unused:**
- Add basic functions as above for future extensibility
- Add docstrings explaining YAML handling
- Import and use config_loader.load_config() as primary source
1. **Update imports:**
- Replace any `from dotenv import load_dotenv` with `from backend.config_loader import load_config, get_config`
- Remove `load_dotenv()` calls
2. **Update main startup:**
- Call `load_config()` at app startup (or rely on module-level auto-run)
- Retrieve config with `get_config()` instead of `os.getenv()`
- Example: `config = get_config()` then `db_path = config['database']['sqlite_path']`
3. **Update environment variable access:**
- Replace `os.getenv("BACKEND_PORT")` with `config.get("backend_port")`
- Replace `os.getenv("JWT_SECRET_KEY")` with `config.get("jwt_secret_key")`
- All refs should come from config dict, not os.getenv()
4. **Remove inventory.env references:**
- Delete any checks for inventory.env existence
- Delete fallback logic to root-level config
- Ensure NO hardcoded "inventory.env" strings remain
5. **Logging:**
- Log at startup which config was loaded (already done by config_loader, but confirm)
- Example: "Backend initialized with config from config/backend.yaml"
Note: This should be minimal changes if main.py already calls config_loader.load_config() at startup.
1. **If entrypoint sources config:**
- Remove any sourcing of inventory.env
- Add comment: "Config is loaded from /app/config/ (YAML format) per Phase 7"
- Ensure /app/config/ path is correct (mapped from host config/ via docker-compose.yml line 19)
2. **Update environment variable documentation:**
- Add comment: "Environment variables override YAML config (D-06 load order)"
- List key variables that can be overridden: JWT_SECRET_KEY, PRIMARY_AI_PROVIDER, etc.
- Example: `export JWT_SECRET_KEY="$(openssl rand -hex 32)"` (if not set)
3. **Ensure startup doesn't fail if config missing:**
- Add check: if config/backend.yaml not found, log error and instructions
- Python code will raise ConfigError, so entrypoint can remain simple
- Just ensure permissions are correct: `chmod 644 config/*.yaml`
4. **Update Dockerfile comment (if present):**
- Reference config/ volume mount
- Explain YAML loading approach
<threat_model>
Trust Boundaries
| Boundary | Description |
|---|---|
| Filesystem → Backend | Backend reads from config/backend.yaml and config/secrets.yaml |
| Environment → Backend | System environment variables override config files (untrusted if exposed in logs) |
| Network → Backend | API receives JWT which is read from config (must be kept secret) |
STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|---|---|---|---|---|
| T-07-05 | Information Disclosure | Config logging | mitigate | Log config values WITHOUT secrets. config_loader.py logs final config but masks JWT_SECRET_KEY, API keys. Pattern: log keys but not values for sensitive fields. |
| T-07-06 | Denial of Service | Invalid YAML parsing | mitigate | yaml.safe_load() prevents code injection. ConfigError raised with clear message if required vars missing. validate_config() checks all required fields. |
| T-07-07 | Tampering | Environment variables | mitigate | Log env var overrides so operators know what was applied. Env vars documented in config/README.md. |
| T-07-08 | Elevation of Privilege | Backend startup | mitigate | ConfigError on missing JWT_SECRET_KEY prevents insecure defaults. Backend refuses to start without proper config. |
</threat_model>
**Phase 7, Plan 2 Verification Checklist:**-
config_loader.py Refactoring
- PyYAML is imported, dotenv is removed
- load_config() loads config/backend.yaml
- load_config() loads config/secrets.yaml (optional)
- Environment variable overrides are applied correctly (D-06)
- validate_config() function checks required variables
- No inventory.env references remain
- Logging shows which config source was used
- File is valid Python syntax
-
config_manager.py Updates
- PyYAML is imported
- read_config() and update_config() functions exist
- YAML file operations use safe_load/safe_dump
- No dotenv references
- File is valid Python syntax
-
main.py Updates
- Imports from backend.config_loader
- Calls load_config() or relies on module-level auto-run
- Uses get_config() to retrieve configuration
- No dotenv or inventory.env references
- No direct os.getenv() calls for app config
- File is valid Python syntax
-
entrypoint.sh Updates
- References config/ paths (not inventory.env)
- Documents environment variable override behavior
- Bash syntax is valid
-
Load Order Verification (D-06)
- System env vars > config/backend.yaml > config/secrets.yaml > defaults
- Integration test: set BACKEND_LOG_LEVEL=DEBUG, verify backend uses DEBUG level
- Integration test: remove config/backend.yaml, verify defaults are used
- Integration test: set JWT_SECRET_KEY in env, verify it overrides YAML value
-
Deprecation Verification (D-04)
- inventory.env no longer used by backend
- No fallback code remains
- Backend fails clearly (ConfigError) if required config missing (instead of silent defaults)
<success_criteria>
- backend/config_loader.py refactored to use PyYAML with env var override support (D-06)
- D-06 load order implemented: env vars > YAML > defaults
- All inventory.env references removed from backend code (D-04)
- config_manager.py updated for YAML file operations
- backend/main.py uses new config loader
- backend/entrypoint.sh references config/ paths
- Configuration validation ensures required variables are present
- Logging shows which sources were used for debugging
- Backend starts successfully with new config structure </success_criteria>