422 lines
18 KiB
Markdown
422 lines
18 KiB
Markdown
---
|
|
phase: 07-config-consolidation
|
|
plan: 02
|
|
type: execute
|
|
wave: 2
|
|
depends_on:
|
|
- 07-01
|
|
files_modified:
|
|
- backend/config_loader.py
|
|
- backend/config_manager.py
|
|
- backend/main.py
|
|
- backend/entrypoint.sh
|
|
autonomous: true
|
|
requirements:
|
|
- PHASE-7-BACKEND-YAML
|
|
- PHASE-7-ENV-OVERRIDE
|
|
- PHASE-7-NO-FALLBACK
|
|
user_setup: []
|
|
|
|
must_haves:
|
|
truths:
|
|
- "Backend loads from config/backend.yaml + config/secrets.yaml (YAML parsing with PyYAML)"
|
|
- "System environment variables override YAML values (per D-06 load order)"
|
|
- "NO fallback to inventory.env—deprecation complete after Phase 7 (per D-04)"
|
|
- "Config loading logs which source is being used for debugging"
|
|
- "Backend starts successfully with new config structure and passes health checks"
|
|
artifacts:
|
|
- path: "backend/config_loader.py"
|
|
provides: "YAML config parsing with env var override and load order enforcement"
|
|
exports: ["load_config()", "get_config()", "validate_config()"]
|
|
min_lines: 80
|
|
- path: "backend/config_manager.py"
|
|
provides: "Config management and updates with YAML support"
|
|
min_lines: 50
|
|
- path: "backend/main.py"
|
|
provides: "Updated main() to use new config_loader (no inventory.env references)"
|
|
pattern: "from backend.config_loader import load_config"
|
|
- path: "backend/entrypoint.sh"
|
|
provides: "Updated Docker entrypoint sourcing YAML config paths"
|
|
pattern: "config/backend.yaml"
|
|
key_links:
|
|
- from: "backend/config_loader.py"
|
|
to: "config/backend.yaml"
|
|
via: "PyYAML parsing"
|
|
pattern: "yaml\\.safe_load.*backend\\.yaml"
|
|
- from: "backend/config_loader.py"
|
|
to: "config/secrets.yaml"
|
|
via: "PyYAML parsing with file existence check"
|
|
pattern: "yaml\\.safe_load.*secrets\\.yaml"
|
|
- from: "backend/main.py"
|
|
to: "backend/config_loader.py"
|
|
via: "import and call load_config()"
|
|
pattern: "from backend.config_loader import load_config"
|
|
---
|
|
|
|
<objective>
|
|
Refactor backend configuration loading from .env to YAML (backend.yaml + secrets.yaml) with system environment variable override support. Remove all inventory.env fallback paths and ensure deprecation is complete.
|
|
|
|
Purpose: Implement D-06 load order (env vars > YAML > defaults) with proper logging and validation.
|
|
|
|
Output: Updated config_loader.py with YAML parsing, config_manager.py with YAML support, main.py using new loader, and updated Docker entrypoint.
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/PROJECT.md
|
|
@.planning/ROADMAP.md
|
|
@.planning/phases/07-config-consolidation/07-CONTEXT.md
|
|
@PROJECT_ARCHITECTURE.md
|
|
@backend/config_loader.py
|
|
@backend/config_manager.py
|
|
@backend/main.py
|
|
@backend/entrypoint.sh
|
|
@config/backend.yaml.example
|
|
@config/secrets.yaml.example
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto">
|
|
<name>Task 1: Refactor backend/config_loader.py for YAML parsing with env var override</name>
|
|
<files>
|
|
backend/config_loader.py
|
|
</files>
|
|
<read_first>
|
|
- backend/config_loader.py (current implementation using dotenv)
|
|
- config/backend.yaml.example (schema to parse)
|
|
- config/secrets.yaml.example (secrets schema)
|
|
- backend/main.py (to understand how config is used)
|
|
</read_first>
|
|
<action>
|
|
Refactor backend/config_loader.py to implement D-06 load order: system env vars > config/backend.yaml > config/secrets.yaml > defaults.
|
|
|
|
**Required changes:**
|
|
|
|
1. **Replace dotenv with PyYAML:** Add `import yaml` and remove `from dotenv import load_dotenv`
|
|
|
|
2. **Implement load_config() function** that:
|
|
- Locates config/ folder (one level up from backend/)
|
|
- Attempts to load config/backend.yaml (required if exists)
|
|
- Attempts to load config/secrets.yaml (optional, file may not exist)
|
|
- Defines hard defaults for all variables (fallback if YAML missing)
|
|
- Merges in order: defaults <- YAML values <- environment variable overrides
|
|
- Returns a dict/object with all configuration
|
|
|
|
3. **Environment variable override pattern:**
|
|
- System env var takes precedence over YAML
|
|
- Naming convention: BACKEND_<YAML_KEY> or just <KEY>
|
|
- Examples:
|
|
- BACKEND_PRIMARY_AI_PROVIDER -> backend.yaml:primary_ai_provider
|
|
- JWT_SECRET_KEY (from secrets.yaml or env)
|
|
- LOG_LEVEL -> backend.yaml:log_level
|
|
- All env vars checked with os.getenv()
|
|
|
|
4. **Load order example (pseudocode):**
|
|
```
|
|
defaults = {primary_ai_provider: "gemini", log_level: "INFO", ...}
|
|
backend_yaml = yaml.safe_load(open("config/backend.yaml")) if exists else {}
|
|
secrets_yaml = yaml.safe_load(open("config/secrets.yaml")) if exists else {}
|
|
config = merge(defaults, backend_yaml, secrets_yaml)
|
|
config = merge(config, env_var_overrides())
|
|
return config
|
|
```
|
|
|
|
5. **Implement validate_config() function** that:
|
|
- Checks required variables are present (JWT_SECRET_KEY, primary_ai_provider, etc.)
|
|
- Validates enum values (primary_ai_provider must be gemini|claude|fallback)
|
|
- Validates log levels (DEBUG|INFO|WARNING|ERROR)
|
|
- Raises ConfigError if validation fails with descriptive message
|
|
|
|
6. **Implement get_config() function** that:
|
|
- Returns the loaded configuration dict
|
|
- Allows other modules to import: `from backend.config_loader import get_config`
|
|
|
|
7. **Logging:**
|
|
- Log which files were loaded: "Loaded backend.yaml from config/"
|
|
- Log env var overrides: "Override primary_ai_provider from environment: gemini"
|
|
- Log final validated config (without secrets): "Config validated: primary_ai_provider=gemini, log_level=INFO"
|
|
- Use log.info() and log.warning() (not print)
|
|
|
|
8. **Remove all inventory.env references:**
|
|
- Delete any checks for inventory_env_path
|
|
- Delete fallback to backend/.env
|
|
- NO fallback to old locations (per D-04 deprecation)
|
|
|
|
9. **Error handling:**
|
|
- If config/backend.yaml missing: raise ConfigError with instructions to copy from .example
|
|
- If secrets.yaml missing: log warning but continue (secrets can come from env vars)
|
|
- If required variables missing after all sources: raise ConfigError listing missing vars
|
|
|
|
10. **Function signature** (updated):
|
|
```python
|
|
def load_config() -> dict:
|
|
"""Load config from YAML files with env var overrides (D-06 load order)."""
|
|
|
|
def get_config() -> dict:
|
|
"""Get loaded config."""
|
|
|
|
def validate_config(config: dict) -> bool:
|
|
"""Validate config has all required values."""
|
|
```
|
|
|
|
Keep the auto-run at module load: `load_config()` and `validate_config()` called on import.
|
|
</action>
|
|
<verify>
|
|
- `grep -q "import yaml" backend/config_loader.py` (PyYAML imported)
|
|
- `grep -q "def load_config" backend/config_loader.py && grep -q "def get_config" backend/config_loader.py` (required functions exist)
|
|
- `grep -q "config/backend.yaml" backend/config_loader.py` (reads backend YAML)
|
|
- `grep -q "config/secrets.yaml" backend/config_loader.py` (reads secrets YAML)
|
|
- `grep -q "os.getenv" backend/config_loader.py` (env var overrides present)
|
|
- `grep -v "inventory.env" backend/config_loader.py | grep -q "inventory"` should return empty (no inventory.env references)
|
|
- `grep -q "validate_config" backend/config_loader.py` (validation function present)
|
|
- `grep -q "log.info\|log.warning" backend/config_loader.py` (logging present)
|
|
- File should be valid Python: `python3 -m py_compile backend/config_loader.py`
|
|
</verify>
|
|
<done>
|
|
config_loader.py refactored to parse YAML files with env var override, remove inventory.env completely, implement D-06 load order with validation and logging.
|
|
</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 2: Update backend/config_manager.py for YAML config updates</name>
|
|
<files>
|
|
backend/config_manager.py
|
|
</files>
|
|
<read_first>
|
|
- backend/config_manager.py (current implementation)
|
|
- config/backend.yaml.example (schema)
|
|
- backend/config_loader.py (just updated)
|
|
</read_first>
|
|
<action>
|
|
Update backend/config_manager.py to support YAML config file updates (if runtime updates are needed):
|
|
|
|
**If config_manager.py currently reads/writes .env files:**
|
|
|
|
1. **Replace dotenv with PyYAML:**
|
|
- Add `import yaml`
|
|
- Remove any dotenv usage
|
|
|
|
2. **Implement update_config() function** that:
|
|
- Takes key-value pairs to update
|
|
- Loads current config/backend.yaml
|
|
- Updates values in-memory
|
|
- Writes back to config/backend.yaml with safe_dump()
|
|
- NEVER writes to config/secrets.yaml (secrets are git-ignored for a reason)
|
|
- Logs what was updated
|
|
|
|
3. **Implement read_config() function** that:
|
|
- Reads config/backend.yaml and returns dict
|
|
- Uses yaml.safe_load()
|
|
|
|
4. **Handle errors gracefully:**
|
|
- If config/backend.yaml not found, raise error (it should exist from Plan 1)
|
|
- If YAML syntax error, log and return current in-memory config
|
|
- Preserve file comments if possible (or warn user they will be lost)
|
|
|
|
5. **Function signature** (updated):
|
|
```python
|
|
def read_config() -> dict:
|
|
"""Read backend.yaml and return current config."""
|
|
|
|
def update_config(updates: dict) -> dict:
|
|
"""Update backend.yaml with new values and return updated config."""
|
|
|
|
def validate_config_file() -> bool:
|
|
"""Validate backend.yaml syntax and required fields."""
|
|
```
|
|
|
|
**If config_manager.py is minimal/unused:**
|
|
- Add basic functions as above for future extensibility
|
|
- Add docstrings explaining YAML handling
|
|
- Import and use config_loader.load_config() as primary source
|
|
</action>
|
|
<verify>
|
|
- `grep -q "import yaml" backend/config_manager.py` (PyYAML imported)
|
|
- `grep -q "def read_config\|def update_config\|def validate_config_file" backend/config_manager.py` (functions present)
|
|
- `grep -q "config/backend.yaml" backend/config_manager.py` (references YAML file)
|
|
- `grep -q "yaml.safe_load\|yaml.safe_dump" backend/config_manager.py` (YAML parsing present)
|
|
- File should be valid Python: `python3 -m py_compile backend/config_manager.py`
|
|
</verify>
|
|
<done>
|
|
config_manager.py updated to support YAML config file updates with safe_load/safe_dump, no dotenv dependencies.
|
|
</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 3: Update backend/main.py to use new YAML config loader</name>
|
|
<files>
|
|
backend/main.py
|
|
</files>
|
|
<read_first>
|
|
- backend/main.py (current implementation)
|
|
- backend/config_loader.py (just updated)
|
|
</read_first>
|
|
<action>
|
|
Update backend/main.py to use the refactored config_loader:
|
|
|
|
1. **Update imports:**
|
|
- Replace any `from dotenv import load_dotenv` with `from backend.config_loader import load_config, get_config`
|
|
- Remove `load_dotenv()` calls
|
|
|
|
2. **Update main startup:**
|
|
- Call `load_config()` at app startup (or rely on module-level auto-run)
|
|
- Retrieve config with `get_config()` instead of `os.getenv()`
|
|
- Example: `config = get_config()` then `db_path = config['database']['sqlite_path']`
|
|
|
|
3. **Update environment variable access:**
|
|
- Replace `os.getenv("BACKEND_PORT")` with `config.get("backend_port")`
|
|
- Replace `os.getenv("JWT_SECRET_KEY")` with `config.get("jwt_secret_key")`
|
|
- All refs should come from config dict, not os.getenv()
|
|
|
|
4. **Remove inventory.env references:**
|
|
- Delete any checks for inventory.env existence
|
|
- Delete fallback logic to root-level config
|
|
- Ensure NO hardcoded "inventory.env" strings remain
|
|
|
|
5. **Logging:**
|
|
- Log at startup which config was loaded (already done by config_loader, but confirm)
|
|
- Example: "Backend initialized with config from config/backend.yaml"
|
|
|
|
Note: This should be minimal changes if main.py already calls config_loader.load_config() at startup.
|
|
</action>
|
|
<verify>
|
|
- `grep -q "from backend.config_loader import" backend/main.py` (imports from new loader)
|
|
- `grep -q "load_dotenv" backend/main.py` should return empty (no dotenv)
|
|
- `grep -q "inventory.env" backend/main.py` should return empty (no old config refs)
|
|
- File should be valid Python: `python3 -m py_compile backend/main.py`
|
|
- Check for os.getenv() calls and ensure they reference config dict instead: `grep "os.getenv" backend/main.py | head -5` (should be minimal or zero)
|
|
</verify>
|
|
<done>
|
|
backend/main.py updated to import and use new YAML-based config_loader, remove all inventory.env and dotenv references.
|
|
</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 4: Update backend/entrypoint.sh for new config paths</name>
|
|
<files>
|
|
backend/entrypoint.sh
|
|
</files>
|
|
<read_first>
|
|
- backend/entrypoint.sh (current Docker entrypoint)
|
|
- docker-compose.yml (volumes mapping config/)
|
|
</read_first>
|
|
<action>
|
|
Update backend/entrypoint.sh to reference new config/ folder paths (per D-07 Docker support):
|
|
|
|
1. **If entrypoint sources config:**
|
|
- Remove any sourcing of inventory.env
|
|
- Add comment: "Config is loaded from /app/config/ (YAML format) per Phase 7"
|
|
- Ensure /app/config/ path is correct (mapped from host config/ via docker-compose.yml line 19)
|
|
|
|
2. **Update environment variable documentation:**
|
|
- Add comment: "Environment variables override YAML config (D-06 load order)"
|
|
- List key variables that can be overridden: JWT_SECRET_KEY, PRIMARY_AI_PROVIDER, etc.
|
|
- Example: `export JWT_SECRET_KEY="$(openssl rand -hex 32)"` (if not set)
|
|
|
|
3. **Ensure startup doesn't fail if config missing:**
|
|
- Add check: if config/backend.yaml not found, log error and instructions
|
|
- Python code will raise ConfigError, so entrypoint can remain simple
|
|
- Just ensure permissions are correct: `chmod 644 config/*.yaml`
|
|
|
|
4. **Update Dockerfile comment (if present):**
|
|
- Reference config/ volume mount
|
|
- Explain YAML loading approach
|
|
</action>
|
|
<verify>
|
|
- `grep -q "config/backend.yaml\|config/secrets.yaml" backend/entrypoint.sh || echo "pass"` (references new config paths or doesn't source at all)
|
|
- `grep -q "inventory.env" backend/entrypoint.sh` should return empty (no old config)
|
|
- File should be valid bash: `bash -n backend/entrypoint.sh`
|
|
</verify>
|
|
<done>
|
|
backend/entrypoint.sh updated to reference config/ paths and document environment variable override behavior.
|
|
</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<threat_model>
|
|
## Trust Boundaries
|
|
|
|
| Boundary | Description |
|
|
|----------|-------------|
|
|
| Filesystem → Backend | Backend reads from config/backend.yaml and config/secrets.yaml |
|
|
| Environment → Backend | System environment variables override config files (untrusted if exposed in logs) |
|
|
| Network → Backend | API receives JWT which is read from config (must be kept secret) |
|
|
|
|
## STRIDE Threat Register
|
|
|
|
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|
|
|-----------|----------|-----------|-------------|-----------------|
|
|
| T-07-05 | Information Disclosure | Config logging | mitigate | Log config values WITHOUT secrets. config_loader.py logs final config but masks JWT_SECRET_KEY, API keys. Pattern: log keys but not values for sensitive fields. |
|
|
| T-07-06 | Denial of Service | Invalid YAML parsing | mitigate | yaml.safe_load() prevents code injection. ConfigError raised with clear message if required vars missing. validate_config() checks all required fields. |
|
|
| T-07-07 | Tampering | Environment variables | mitigate | Log env var overrides so operators know what was applied. Env vars documented in config/README.md. |
|
|
| T-07-08 | Elevation of Privilege | Backend startup | mitigate | ConfigError on missing JWT_SECRET_KEY prevents insecure defaults. Backend refuses to start without proper config. |
|
|
|
|
</threat_model>
|
|
|
|
<verification>
|
|
**Phase 7, Plan 2 Verification Checklist:**
|
|
|
|
1. **config_loader.py Refactoring**
|
|
- [ ] PyYAML is imported, dotenv is removed
|
|
- [ ] load_config() loads config/backend.yaml
|
|
- [ ] load_config() loads config/secrets.yaml (optional)
|
|
- [ ] Environment variable overrides are applied correctly (D-06)
|
|
- [ ] validate_config() function checks required variables
|
|
- [ ] No inventory.env references remain
|
|
- [ ] Logging shows which config source was used
|
|
- [ ] File is valid Python syntax
|
|
|
|
2. **config_manager.py Updates**
|
|
- [ ] PyYAML is imported
|
|
- [ ] read_config() and update_config() functions exist
|
|
- [ ] YAML file operations use safe_load/safe_dump
|
|
- [ ] No dotenv references
|
|
- [ ] File is valid Python syntax
|
|
|
|
3. **main.py Updates**
|
|
- [ ] Imports from backend.config_loader
|
|
- [ ] Calls load_config() or relies on module-level auto-run
|
|
- [ ] Uses get_config() to retrieve configuration
|
|
- [ ] No dotenv or inventory.env references
|
|
- [ ] No direct os.getenv() calls for app config
|
|
- [ ] File is valid Python syntax
|
|
|
|
4. **entrypoint.sh Updates**
|
|
- [ ] References config/ paths (not inventory.env)
|
|
- [ ] Documents environment variable override behavior
|
|
- [ ] Bash syntax is valid
|
|
|
|
5. **Load Order Verification (D-06)**
|
|
- [ ] System env vars > config/backend.yaml > config/secrets.yaml > defaults
|
|
- [ ] Integration test: set BACKEND_LOG_LEVEL=DEBUG, verify backend uses DEBUG level
|
|
- [ ] Integration test: remove config/backend.yaml, verify defaults are used
|
|
- [ ] Integration test: set JWT_SECRET_KEY in env, verify it overrides YAML value
|
|
|
|
6. **Deprecation Verification (D-04)**
|
|
- [ ] inventory.env no longer used by backend
|
|
- [ ] No fallback code remains
|
|
- [ ] Backend fails clearly (ConfigError) if required config missing (instead of silent defaults)
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- backend/config_loader.py refactored to use PyYAML with env var override support (D-06)
|
|
- D-06 load order implemented: env vars > YAML > defaults
|
|
- All inventory.env references removed from backend code (D-04)
|
|
- config_manager.py updated for YAML file operations
|
|
- backend/main.py uses new config loader
|
|
- backend/entrypoint.sh references config/ paths
|
|
- Configuration validation ensures required variables are present
|
|
- Logging shows which sources were used for debugging
|
|
- Backend starts successfully with new config structure
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/07-config-consolidation/07-02-SUMMARY.md`
|
|
</output>
|