bede/tfm_ainventory
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(phase-7): create detailed phase plans for config consolidation (4 plans, 14 tasks)

Browse Source
This commit is contained in:
Daniel Bedeleanu
2026-04-23 12:29:05 +03:00
parent dacf7d2921
commit b7218175bb
4 changed files with 2041 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

395
.planning/phases/07-config-consolidation/07-01-PLAN.md Normal file
View File

@@ -0,0 +1,395 @@
---
phase: 07-config-consolidation
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- config/backend.yaml
- config/backend.yaml.example
- config/frontend.yaml
- config/frontend.yaml.example
- config/network.yaml
- config/network.yaml.example
- config/docker.yaml
- config/docker.yaml.example
- config/secrets.yaml.example
- config/README.md
- .gitignore
autonomous: true
requirements:
- PHASE-7-CONFIG-STRUCT
- PHASE-7-YAML-FORMAT
- PHASE-7-SECRETS-MGMT
- PHASE-7-DOCUMENTATION
user_setup: []
must_haves:
truths:
- "config/ folder exists with all YAML files and examples"
- "YAML structure matches backend, frontend, network, docker, and secrets domains"
- "All *.yaml.example files committed to git showing complete schema"
- "config/secrets.yaml is git-ignored, with example provided"
- ".gitignore correctly tracks examples, ignores actual secrets"
- "config/README.md documents every YAML file with required variables and setup instructions"
artifacts:
- path: "config/backend.yaml.example"
provides: "Backend configuration schema (database, AI keys, auth, logging)"
min_lines: 40
- path: "config/frontend.yaml.example"
provides: "Frontend configuration schema (API endpoints, feature flags, PWA)"
min_lines: 25
- path: "config/network.yaml.example"
provides: "Network configuration schema (ports, SSL, CORS, server IPs)"
min_lines: 20
- path: "config/docker.yaml.example"
provides: "Docker-specific overrides (container resources, mount paths)"
min_lines: 20
- path: "config/secrets.yaml.example"
provides: "Secrets template (JWT, API keys, passwords)"
min_lines: 15
- path: "config/README.md"
provides: "Comprehensive documentation of all config files and setup"
min_lines: 100
key_links:
- from: "config/backend.yaml"
to: "backend/config_loader.py"
via: "YAML parsing in config loader"
pattern: "load.*backend\\.yaml"
- from: "config/secrets.yaml.example"
to: ".gitignore"
via: "git ignore rule"
pattern: "config/\\*\\.yaml.*!.*\\.example"
- from: "config/"
to: "docker-compose.yml"
via: "volume mount in service definition"
pattern: "\\./config:/app/config"
---
<objective>
Create the config/ folder structure with YAML files for backend, frontend, network, docker, and secrets configurations. Establish the foundation for centralized configuration management with clear schema documentation.
Purpose: Consolidate scattered configuration (currently in inventory.env) into a structured, domain-specific YAML format (per D-01, D-02, D-03).
Output: config/ folder with 4 YAML config files + examples, secrets template, comprehensive README, and .gitignore updates.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/07-config-consolidation/07-CONTEXT.md
@PROJECT_ARCHITECTURE.md
@DEPLOYMENT.md
@inventory.env
@inventory.env.example
@docker-compose.yml
</context>
<tasks>
<task type="auto">
<name>Task 1: Create config/ folder structure and YAML example files</name>
<files>
config/backend.yaml.example
config/frontend.yaml.example
config/network.yaml.example
config/docker.yaml.example
config/secrets.yaml.example
</files>
<read_first>
- inventory.env (current config source)
- inventory.env.example (existing schema)
- PROJECT_ARCHITECTURE.md (tech stack, components)
- DEPLOYMENT.md (current config categories)
- docker-compose.yml (container environment vars)
</read_first>
<action>
Create config/ folder in project root with 5 YAML example files (per D-01, D-03):
**config/backend.yaml.example** — Backend configuration template covering:
- Database: sqlite_path, log_retention_days, wal_mode (from current code patterns)
- AI: primary_ai_provider (gemini|claude), gemini_api_key, claude_api_key, fallback_provider
- Auth: jwt_secret_key, ldap_server (optional), ldap_base_dn, password_cache_path
- Logging: log_level (DEBUG|INFO|WARNING|ERROR), log_rotation_size_mb, log_rotation_count
- Application: data_dir, logs_dir, cors_origins
- Feature flags: ai_extraction_enabled, offline_sync_enabled, audit_logging_enabled
**config/frontend.yaml.example** — Frontend configuration template covering:
- API: backend_url (e.g., http://localhost:8916), timeout_ms
- Feature flags: service_worker_enabled, offline_enabled, ai_extraction_ui_enabled
- PWA: app_name, short_name, start_url, display_mode
- Feature toggles: enable_qr_scanner, enable_barcode_scanner, enable_batch_import
**config/network.yaml.example** — Network/deployment configuration template covering:
- Ports: backend_port (8916), frontend_port (8917), backend_ssl_port (8918), frontend_ssl_port (8919)
- SSL: ssl_enabled (true|false), certificate_path, key_path (or auto-generated by Caddy)
- Proxy: caddy_log_level, proxy_read_timeout_s, max_request_size_mb
- CORS: allowed_origins (comma-separated), allowed_methods, allowed_headers
**config/docker.yaml.example** — Docker-specific overrides covering:
- Images: backend_image, frontend_image, proxy_image (with tags)
- Resources: backend_cpu_limit, backend_memory_limit, frontend_cpu_limit, frontend_memory_limit
- Volumes: data_volume_driver, logs_volume_driver, use_named_volumes (true|false)
- Networks: network_name, network_driver (bridge|overlay)
**config/secrets.yaml.example** — Secrets template (git-ignored) covering:
- JWT_SECRET_KEY: "CHANGE_ME_IN_PRODUCTION" (minimum 32 chars)
- GEMINI_API_KEY: "your-api-key"
- CLAUDE_API_KEY: "your-api-key"
- DATABASE_PASSWORD: "db-password" (if using external DB)
- LDAP_PASSWORD: "ldap-password" (if using LDAP)
- CORS_ORIGIN_PASSWORD: (if CORS origins require auth)
All examples should have comments explaining:
- What each variable controls
- Default values
- Valid value ranges
- Where to obtain secrets (e.g., "Generate with: openssl rand -hex 32")
- Whether the variable is required or optional
</action>
<verify>
- `test -d config` (folder exists)
- `test -f config/backend.yaml.example && grep -q "primary_ai_provider" config/backend.yaml.example` (backend schema present)
- `test -f config/frontend.yaml.example && grep -q "backend_url" config/frontend.yaml.example` (frontend schema present)
- `test -f config/network.yaml.example && grep -q "backend_port" config/network.yaml.example` (network schema present)
- `test -f config/docker.yaml.example && grep -q "backend_cpu_limit" config/docker.yaml.example` (docker schema present)
- `test -f config/secrets.yaml.example && grep -q "JWT_SECRET_KEY" config/secrets.yaml.example` (secrets template present)
- All 5 files are readable and valid YAML syntax: `python3 -c "import yaml; [yaml.safe_load(open(f)) for f in ['config/backend.yaml.example', 'config/frontend.yaml.example', 'config/network.yaml.example', 'config/docker.yaml.example', 'config/secrets.yaml.example']]"`
</verify>
<done>
config/ folder created with 5 YAML example files, all valid YAML syntax, comprehensive comments documenting schema, required/optional status, and secret generation instructions.
</done>
</task>
<task type="auto">
<name>Task 2: Create actual config files from examples and establish .gitignore rules</name>
<files>
config/backend.yaml
config/frontend.yaml
config/network.yaml
config/docker.yaml
.gitignore
</files>
<read_first>
- config/backend.yaml.example (just created)
- config/frontend.yaml.example (just created)
- config/network.yaml.example (just created)
- config/docker.yaml.example (just created)
- inventory.env (current actual values to migrate)
- .gitignore (current ignore rules)
</read_first>
<action>
Create actual (non-example) YAML config files by copying examples and filling in values from inventory.env (per D-03 pattern):
**config/backend.yaml** — Copy from backend.yaml.example and fill in values from inventory.env:
- primary_ai_provider: (from PRIMARY_AI_PROVIDER in inventory.env, default: gemini)
- gemini_api_key: (from GEMINI_API_KEY if present)
- claude_api_key: (from CLAUDE_API_KEY if present)
- ldap_server: (from LDAP_SERVER if present, or empty)
- log_level: (from LOG_LEVEL if present, default: INFO)
- data_dir: ./data
- logs_dir: ./logs
- cors_origins: (from CORS_ORIGINS if present, or default: http://localhost:8917)
**config/frontend.yaml** — Create from frontend.yaml.example:
- backend_url: (from BACKEND_URL or derived from BACKEND_PORT, default: http://localhost:8916)
- timeout_ms: 30000
- service_worker_enabled: true
- offline_enabled: true
- ai_extraction_ui_enabled: true
**config/network.yaml** — Create from network.yaml.example:
- backend_port: (from BACKEND_PORT in inventory.env, default: 8916)
- frontend_port: (from FRONTEND_PORT in inventory.env, default: 8917)
- backend_ssl_port: (from BACKEND_SSL_PORT if present, default: 8918)
- frontend_ssl_port: (from FRONTEND_SSL_PORT if present, default: 8919)
- ssl_enabled: true
- allowed_origins: (from CORS_ORIGINS if present)
**config/docker.yaml** — Create from docker.yaml.example:
- backend_cpu_limit: "1.0"
- backend_memory_limit: "1G"
- frontend_cpu_limit: "0.5"
- frontend_memory_limit: "512M"
- use_named_volumes: true
**Update .gitignore** (per D-08):
Add rules to IGNORE actual config files (secrets protection) but TRACK examples:
```
# Config files — ignore all .yaml except examples
config/*.yaml
!config/*.yaml.example
config/secrets.yaml
!config/secrets.yaml.example
```
Do NOT delete old inventory.env yet (backward compatibility until Phase 7 complete, per D-04 deprecation timeline).
</action>
<verify>
- `test -f config/backend.yaml && grep -q "primary_ai_provider" config/backend.yaml` (backend.yaml exists and has content)
- `test -f config/frontend.yaml && grep -q "backend_url" config/frontend.yaml` (frontend.yaml exists)
- `test -f config/network.yaml && grep -q "backend_port" config/network.yaml` (network.yaml exists)
- `test -f config/docker.yaml && grep -q "backend_cpu_limit" config/docker.yaml` (docker.yaml exists)
- All 4 files are valid YAML: `python3 -c "import yaml; [yaml.safe_load(open(f)) for f in ['config/backend.yaml', 'config/frontend.yaml', 'config/network.yaml', 'config/docker.yaml']]"`
- .gitignore contains rules: `grep -q "config/\*\.yaml" .gitignore && grep -q "!config/\*\.yaml\.example" .gitignore`
- config/secrets.yaml does NOT exist (will be created manually by developers from example)
</verify>
<done>
Actual config files created with production values migrated from inventory.env, .gitignore updated to track examples and ignore actual config files (including secrets).
</done>
</task>
<task type="auto">
<name>Task 3: Create comprehensive config/README.md documentation</name>
<files>
config/README.md
</files>
<read_first>
- config/backend.yaml.example
- config/frontend.yaml.example
- config/network.yaml.example
- config/docker.yaml.example
- config/secrets.yaml.example
- DEPLOYMENT.md (current deployment docs)
</read_first>
<action>
Create config/README.md with comprehensive documentation (per D-08) covering:
**Section 1: Overview**
- Explain that config/ is the single source of truth for application configuration
- Mention load order: system environment variables > config/*.yaml > defaults in code
- Note that secrets are separate (git-ignored)
**Section 2: Quick Start**
- Copy all *.example files to remove .example suffix
- Fill in required values (especially JWT_SECRET_KEY, API keys)
- Create secrets.yaml from secrets.yaml.example with actual values
**Section 3: backend.yaml**
- Explain each variable: purpose, valid values, defaults, required/optional
- List how to obtain each value (e.g., "Generate JWT key with: openssl rand -hex 32")
- Show example values
- List environment variable override names (e.g., BACKEND_PRIMARY_AI_PROVIDER)
**Section 4: frontend.yaml**
- Explain each variable: purpose, valid values, defaults
- List required vs optional flags
- Show how feature flags affect UI behavior
- List environment variable override names
**Section 5: network.yaml**
- Explain port assignments and SSL settings
- Show how CORS origins work
- List default values and adjustment guidance
- List environment variable override names
**Section 6: docker.yaml**
- Explain container resource limits (CPU, memory)
- Show how to adjust for different hardware
- Explain volume management
- List environment variable override names
**Section 7: secrets.yaml**
- Explain git-ignore protection
- List all required secrets with generation/obtainment instructions
- Warn about security implications
- Show correct file permissions (600)
**Section 8: Environment Variable Override**
- Explain how system environment variables take precedence over YAML
- Show naming convention (e.g., BACKEND_PRIMARY_AI_PROVIDER -> backend.yaml:primary_ai_provider)
- Useful for Docker deployments where secrets come from docker run -e
**Section 9: Troubleshooting**
- Common issues: missing secrets, invalid YAML syntax, missing required values
- How to validate YAML syntax
- How to debug which config source is being used
Include helpful tables showing all variables, their purposes, defaults, and how to override them.
</action>
<verify>
- `test -f config/README.md && wc -l config/README.md | awk '{print $1}' | awk '$1 >= 100 {print "pass"}'` (README has at least 100 lines)
- `grep -q "Quick Start" config/README.md && grep -q "backend.yaml" config/README.md && grep -q "secrets.yaml" config/README.md` (README covers all files)
- `grep -q "environment variable" config/README.md` (override mechanism documented)
- `grep -q "JWT_SECRET_KEY" config/README.md && grep -q "openssl rand" config/README.md` (secret generation instructions present)
</verify>
<done>
config/README.md created with comprehensive documentation of all YAML files, required variables, generation instructions, environment variable overrides, and troubleshooting guide.
</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Filesystem → Application | Config files loaded from disk must not be tampered with |
| Environment → Application | System environment variables override YAML (untrusted if exposed) |
| Git repository → Deployment | .gitignore must prevent accidental commit of secrets |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-07-01 | Tampering | config/*.yaml | mitigate | File permissions enforced via git (0644 for examples, 0600 for secrets). Read-only volume mounts in Docker per docker-compose.yml line 19 `:ro` flag. |
| T-07-02 | Information Disclosure | config/secrets.yaml | mitigate | .gitignore rule prevents accidental commit. File permissions enforced (chmod 600). config/README.md warns developers about security. Example file included to guide setup. |
| T-07-03 | Denial of Service | config/*.yaml parsing | mitigate | PyYAML used with safe_load() only (no arbitrary code execution). Backend config_loader.py validates syntax before loading. Invalid YAML causes logged error + default fallback per D-06 load order. |
| T-07-04 | Elevation of Privilege | JWT_SECRET_KEY exposure | accept | JWT secret hardcoded in docker-compose.yml example (line 25) warns with comment. Developers must provide production value. Risk low for development environments. |
</threat_model>
<verification>
**Phase 7, Plan 1 Verification Checklist:**
1. **Config Folder Structure**
- [ ] `config/` folder exists in project root
- [ ] 5 example files present: backend.yaml.example, frontend.yaml.example, network.yaml.example, docker.yaml.example, secrets.yaml.example
- [ ] All examples contain valid YAML syntax (parseable by python3 -m yaml)
- [ ] All examples have comprehensive comments explaining each variable
2. **Actual Config Files**
- [ ] 4 actual config files exist: backend.yaml, frontend.yaml, network.yaml, docker.yaml
- [ ] All actual files contain valid YAML syntax
- [ ] Values migrated from inventory.env are present and reasonable
- [ ] secrets.yaml does NOT exist (will be created manually)
3. **Git Integration**
- [ ] .gitignore updated with rules: `config/*.yaml`, `!config/*.yaml.example`, `config/secrets.yaml`, `!config/secrets.yaml.example`
- [ ] Examples are tracked: `git status config/*.example` shows "new file" or no changes
- [ ] Actual configs are ignored: `git check-ignore config/backend.yaml` returns success
- [ ] Secrets example is tracked but secrets themselves are ignored
4. **Documentation**
- [ ] config/README.md exists with 100+ lines
- [ ] README covers all 5 YAML files (backend, frontend, network, docker, secrets)
- [ ] README documents environment variable override mechanism
- [ ] README includes secret generation/obtainment instructions
- [ ] README has troubleshooting section
5. **Backward Compatibility (D-04)**
- [ ] inventory.env still exists (will be fully deprecated after backend refactor in Plan 2)
- [ ] No code changes yet (Config loading still uses inventory.env, Phase 7 Plan 2 updates backend)
</verification>
<success_criteria>
- config/ folder created with 5 YAML example files defining complete schema
- 4 actual YAML config files created with values migrated from inventory.env
- config/secrets.yaml.example provides template (actual secrets.yaml created manually by developers)
- .gitignore updated to track examples, ignore actual config files and secrets
- config/README.md provides comprehensive documentation and setup instructions
- All YAML files are syntactically valid and parseable
- No backend code changes yet (backward compatibility maintained per D-04)
- Foundation ready for backend refactoring in Plan 2
</success_criteria>
<output>
After completion, create `.planning/phases/07-config-consolidation/07-01-SUMMARY.md`
</output>

421
.planning/phases/07-config-consolidation/07-02-PLAN.md Normal file
View File

@@ -0,0 +1,421 @@
---
phase: 07-config-consolidation
plan: 02
type: execute
wave: 2
depends_on:
- 07-01
files_modified:
- backend/config_loader.py
- backend/config_manager.py
- backend/main.py
- backend/entrypoint.sh
autonomous: true
requirements:
- PHASE-7-BACKEND-YAML
- PHASE-7-ENV-OVERRIDE
- PHASE-7-NO-FALLBACK
user_setup: []
must_haves:
truths:
- "Backend loads from config/backend.yaml + config/secrets.yaml (YAML parsing with PyYAML)"
- "System environment variables override YAML values (per D-06 load order)"
- "NO fallback to inventory.env—deprecation complete after Phase 7 (per D-04)"
- "Config loading logs which source is being used for debugging"
- "Backend starts successfully with new config structure and passes health checks"
artifacts:
- path: "backend/config_loader.py"
provides: "YAML config parsing with env var override and load order enforcement"
exports: ["load_config()", "get_config()", "validate_config()"]
min_lines: 80
- path: "backend/config_manager.py"
provides: "Config management and updates with YAML support"
min_lines: 50
- path: "backend/main.py"
provides: "Updated main() to use new config_loader (no inventory.env references)"
pattern: "from backend.config_loader import load_config"
- path: "backend/entrypoint.sh"
provides: "Updated Docker entrypoint sourcing YAML config paths"
pattern: "config/backend.yaml"
key_links:
- from: "backend/config_loader.py"
to: "config/backend.yaml"
via: "PyYAML parsing"
pattern: "yaml\\.safe_load.*backend\\.yaml"
- from: "backend/config_loader.py"
to: "config/secrets.yaml"
via: "PyYAML parsing with file existence check"
pattern: "yaml\\.safe_load.*secrets\\.yaml"
- from: "backend/main.py"
to: "backend/config_loader.py"
via: "import and call load_config()"
pattern: "from backend.config_loader import load_config"
---
<objective>
Refactor backend configuration loading from .env to YAML (backend.yaml + secrets.yaml) with system environment variable override support. Remove all inventory.env fallback paths and ensure deprecation is complete.
Purpose: Implement D-06 load order (env vars > YAML > defaults) with proper logging and validation.
Output: Updated config_loader.py with YAML parsing, config_manager.py with YAML support, main.py using new loader, and updated Docker entrypoint.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/07-config-consolidation/07-CONTEXT.md
@PROJECT_ARCHITECTURE.md
@backend/config_loader.py
@backend/config_manager.py
@backend/main.py
@backend/entrypoint.sh
@config/backend.yaml.example
@config/secrets.yaml.example
</context>
<tasks>
<task type="auto">
<name>Task 1: Refactor backend/config_loader.py for YAML parsing with env var override</name>
<files>
backend/config_loader.py
</files>
<read_first>
- backend/config_loader.py (current implementation using dotenv)
- config/backend.yaml.example (schema to parse)
- config/secrets.yaml.example (secrets schema)
- backend/main.py (to understand how config is used)
</read_first>
<action>
Refactor backend/config_loader.py to implement D-06 load order: system env vars > config/backend.yaml > config/secrets.yaml > defaults.
**Required changes:**
1. **Replace dotenv with PyYAML:** Add `import yaml` and remove `from dotenv import load_dotenv`
2. **Implement load_config() function** that:
- Locates config/ folder (one level up from backend/)
- Attempts to load config/backend.yaml (required if exists)
- Attempts to load config/secrets.yaml (optional, file may not exist)
- Defines hard defaults for all variables (fallback if YAML missing)
- Merges in order: defaults <- YAML values <- environment variable overrides
- Returns a dict/object with all configuration
3. **Environment variable override pattern:**
- System env var takes precedence over YAML
- Naming convention: BACKEND_<YAML_KEY> or just <KEY>
- Examples:
- BACKEND_PRIMARY_AI_PROVIDER -> backend.yaml:primary_ai_provider
- JWT_SECRET_KEY (from secrets.yaml or env)
- LOG_LEVEL -> backend.yaml:log_level
- All env vars checked with os.getenv()
4. **Load order example (pseudocode):**
```
defaults = {primary_ai_provider: "gemini", log_level: "INFO", ...}
backend_yaml = yaml.safe_load(open("config/backend.yaml")) if exists else {}
secrets_yaml = yaml.safe_load(open("config/secrets.yaml")) if exists else {}
config = merge(defaults, backend_yaml, secrets_yaml)
config = merge(config, env_var_overrides())
return config
```
5. **Implement validate_config() function** that:
- Checks required variables are present (JWT_SECRET_KEY, primary_ai_provider, etc.)
- Validates enum values (primary_ai_provider must be gemini|claude|fallback)
- Validates log levels (DEBUG|INFO|WARNING|ERROR)
- Raises ConfigError if validation fails with descriptive message
6. **Implement get_config() function** that:
- Returns the loaded configuration dict
- Allows other modules to import: `from backend.config_loader import get_config`
7. **Logging:**
- Log which files were loaded: "Loaded backend.yaml from config/"
- Log env var overrides: "Override primary_ai_provider from environment: gemini"
- Log final validated config (without secrets): "Config validated: primary_ai_provider=gemini, log_level=INFO"
- Use log.info() and log.warning() (not print)
8. **Remove all inventory.env references:**
- Delete any checks for inventory_env_path
- Delete fallback to backend/.env
- NO fallback to old locations (per D-04 deprecation)
9. **Error handling:**
- If config/backend.yaml missing: raise ConfigError with instructions to copy from .example
- If secrets.yaml missing: log warning but continue (secrets can come from env vars)
- If required variables missing after all sources: raise ConfigError listing missing vars
10. **Function signature** (updated):
```python
def load_config() -> dict:
"""Load config from YAML files with env var overrides (D-06 load order)."""
def get_config() -> dict:
"""Get loaded config."""
def validate_config(config: dict) -> bool:
"""Validate config has all required values."""
```
Keep the auto-run at module load: `load_config()` and `validate_config()` called on import.
</action>
<verify>
- `grep -q "import yaml" backend/config_loader.py` (PyYAML imported)
- `grep -q "def load_config" backend/config_loader.py && grep -q "def get_config" backend/config_loader.py` (required functions exist)
- `grep -q "config/backend.yaml" backend/config_loader.py` (reads backend YAML)
- `grep -q "config/secrets.yaml" backend/config_loader.py` (reads secrets YAML)
- `grep -q "os.getenv" backend/config_loader.py` (env var overrides present)
- `grep -v "inventory.env" backend/config_loader.py | grep -q "inventory"` should return empty (no inventory.env references)
- `grep -q "validate_config" backend/config_loader.py` (validation function present)
- `grep -q "log.info\|log.warning" backend/config_loader.py` (logging present)
- File should be valid Python: `python3 -m py_compile backend/config_loader.py`
</verify>
<done>
config_loader.py refactored to parse YAML files with env var override, remove inventory.env completely, implement D-06 load order with validation and logging.
</done>
</task>
<task type="auto">
<name>Task 2: Update backend/config_manager.py for YAML config updates</name>
<files>
backend/config_manager.py
</files>
<read_first>
- backend/config_manager.py (current implementation)
- config/backend.yaml.example (schema)
- backend/config_loader.py (just updated)
</read_first>
<action>
Update backend/config_manager.py to support YAML config file updates (if runtime updates are needed):
**If config_manager.py currently reads/writes .env files:**
1. **Replace dotenv with PyYAML:**
- Add `import yaml`
- Remove any dotenv usage
2. **Implement update_config() function** that:
- Takes key-value pairs to update
- Loads current config/backend.yaml
- Updates values in-memory
- Writes back to config/backend.yaml with safe_dump()
- NEVER writes to config/secrets.yaml (secrets are git-ignored for a reason)
- Logs what was updated
3. **Implement read_config() function** that:
- Reads config/backend.yaml and returns dict
- Uses yaml.safe_load()
4. **Handle errors gracefully:**
- If config/backend.yaml not found, raise error (it should exist from Plan 1)
- If YAML syntax error, log and return current in-memory config
- Preserve file comments if possible (or warn user they will be lost)
5. **Function signature** (updated):
```python
def read_config() -> dict:
"""Read backend.yaml and return current config."""
def update_config(updates: dict) -> dict:
"""Update backend.yaml with new values and return updated config."""
def validate_config_file() -> bool:
"""Validate backend.yaml syntax and required fields."""
```
**If config_manager.py is minimal/unused:**
- Add basic functions as above for future extensibility
- Add docstrings explaining YAML handling
- Import and use config_loader.load_config() as primary source
</action>
<verify>
- `grep -q "import yaml" backend/config_manager.py` (PyYAML imported)
- `grep -q "def read_config\|def update_config\|def validate_config_file" backend/config_manager.py` (functions present)
- `grep -q "config/backend.yaml" backend/config_manager.py` (references YAML file)
- `grep -q "yaml.safe_load\|yaml.safe_dump" backend/config_manager.py` (YAML parsing present)
- File should be valid Python: `python3 -m py_compile backend/config_manager.py`
</verify>
<done>
config_manager.py updated to support YAML config file updates with safe_load/safe_dump, no dotenv dependencies.
</done>
</task>
<task type="auto">
<name>Task 3: Update backend/main.py to use new YAML config loader</name>
<files>
backend/main.py
</files>
<read_first>
- backend/main.py (current implementation)
- backend/config_loader.py (just updated)
</read_first>
<action>
Update backend/main.py to use the refactored config_loader:
1. **Update imports:**
- Replace any `from dotenv import load_dotenv` with `from backend.config_loader import load_config, get_config`
- Remove `load_dotenv()` calls
2. **Update main startup:**
- Call `load_config()` at app startup (or rely on module-level auto-run)
- Retrieve config with `get_config()` instead of `os.getenv()`
- Example: `config = get_config()` then `db_path = config['database']['sqlite_path']`
3. **Update environment variable access:**
- Replace `os.getenv("BACKEND_PORT")` with `config.get("backend_port")`
- Replace `os.getenv("JWT_SECRET_KEY")` with `config.get("jwt_secret_key")`
- All refs should come from config dict, not os.getenv()
4. **Remove inventory.env references:**
- Delete any checks for inventory.env existence
- Delete fallback logic to root-level config
- Ensure NO hardcoded "inventory.env" strings remain
5. **Logging:**
- Log at startup which config was loaded (already done by config_loader, but confirm)
- Example: "Backend initialized with config from config/backend.yaml"
Note: This should be minimal changes if main.py already calls config_loader.load_config() at startup.
</action>
<verify>
- `grep -q "from backend.config_loader import" backend/main.py` (imports from new loader)
- `grep -q "load_dotenv" backend/main.py` should return empty (no dotenv)
- `grep -q "inventory.env" backend/main.py` should return empty (no old config refs)
- File should be valid Python: `python3 -m py_compile backend/main.py`
- Check for os.getenv() calls and ensure they reference config dict instead: `grep "os.getenv" backend/main.py | head -5` (should be minimal or zero)
</verify>
<done>
backend/main.py updated to import and use new YAML-based config_loader, remove all inventory.env and dotenv references.
</done>
</task>
<task type="auto">
<name>Task 4: Update backend/entrypoint.sh for new config paths</name>
<files>
backend/entrypoint.sh
</files>
<read_first>
- backend/entrypoint.sh (current Docker entrypoint)
- docker-compose.yml (volumes mapping config/)
</read_first>
<action>
Update backend/entrypoint.sh to reference new config/ folder paths (per D-07 Docker support):
1. **If entrypoint sources config:**
- Remove any sourcing of inventory.env
- Add comment: "Config is loaded from /app/config/ (YAML format) per Phase 7"
- Ensure /app/config/ path is correct (mapped from host config/ via docker-compose.yml line 19)
2. **Update environment variable documentation:**
- Add comment: "Environment variables override YAML config (D-06 load order)"
- List key variables that can be overridden: JWT_SECRET_KEY, PRIMARY_AI_PROVIDER, etc.
- Example: `export JWT_SECRET_KEY="$(openssl rand -hex 32)"` (if not set)
3. **Ensure startup doesn't fail if config missing:**
- Add check: if config/backend.yaml not found, log error and instructions
- Python code will raise ConfigError, so entrypoint can remain simple
- Just ensure permissions are correct: `chmod 644 config/*.yaml`
4. **Update Dockerfile comment (if present):**
- Reference config/ volume mount
- Explain YAML loading approach
</action>
<verify>
- `grep -q "config/backend.yaml\|config/secrets.yaml" backend/entrypoint.sh || echo "pass"` (references new config paths or doesn't source at all)
- `grep -q "inventory.env" backend/entrypoint.sh` should return empty (no old config)
- File should be valid bash: `bash -n backend/entrypoint.sh`
</verify>
<done>
backend/entrypoint.sh updated to reference config/ paths and document environment variable override behavior.
</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Filesystem → Backend | Backend reads from config/backend.yaml and config/secrets.yaml |
| Environment → Backend | System environment variables override config files (untrusted if exposed in logs) |
| Network → Backend | API receives JWT which is read from config (must be kept secret) |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-07-05 | Information Disclosure | Config logging | mitigate | Log config values WITHOUT secrets. config_loader.py logs final config but masks JWT_SECRET_KEY, API keys. Pattern: log keys but not values for sensitive fields. |
| T-07-06 | Denial of Service | Invalid YAML parsing | mitigate | yaml.safe_load() prevents code injection. ConfigError raised with clear message if required vars missing. validate_config() checks all required fields. |
| T-07-07 | Tampering | Environment variables | mitigate | Log env var overrides so operators know what was applied. Env vars documented in config/README.md. |
| T-07-08 | Elevation of Privilege | Backend startup | mitigate | ConfigError on missing JWT_SECRET_KEY prevents insecure defaults. Backend refuses to start without proper config. |
</threat_model>
<verification>
**Phase 7, Plan 2 Verification Checklist:**
1. **config_loader.py Refactoring**
- [ ] PyYAML is imported, dotenv is removed
- [ ] load_config() loads config/backend.yaml
- [ ] load_config() loads config/secrets.yaml (optional)
- [ ] Environment variable overrides are applied correctly (D-06)
- [ ] validate_config() function checks required variables
- [ ] No inventory.env references remain
- [ ] Logging shows which config source was used
- [ ] File is valid Python syntax
2. **config_manager.py Updates**
- [ ] PyYAML is imported
- [ ] read_config() and update_config() functions exist
- [ ] YAML file operations use safe_load/safe_dump
- [ ] No dotenv references
- [ ] File is valid Python syntax
3. **main.py Updates**
- [ ] Imports from backend.config_loader
- [ ] Calls load_config() or relies on module-level auto-run
- [ ] Uses get_config() to retrieve configuration
- [ ] No dotenv or inventory.env references
- [ ] No direct os.getenv() calls for app config
- [ ] File is valid Python syntax
4. **entrypoint.sh Updates**
- [ ] References config/ paths (not inventory.env)
- [ ] Documents environment variable override behavior
- [ ] Bash syntax is valid
5. **Load Order Verification (D-06)**
- [ ] System env vars > config/backend.yaml > config/secrets.yaml > defaults
- [ ] Integration test: set BACKEND_LOG_LEVEL=DEBUG, verify backend uses DEBUG level
- [ ] Integration test: remove config/backend.yaml, verify defaults are used
- [ ] Integration test: set JWT_SECRET_KEY in env, verify it overrides YAML value
6. **Deprecation Verification (D-04)**
- [ ] inventory.env no longer used by backend
- [ ] No fallback code remains
- [ ] Backend fails clearly (ConfigError) if required config missing (instead of silent defaults)
</verification>
<success_criteria>
- backend/config_loader.py refactored to use PyYAML with env var override support (D-06)
- D-06 load order implemented: env vars > YAML > defaults
- All inventory.env references removed from backend code (D-04)
- config_manager.py updated for YAML file operations
- backend/main.py uses new config loader
- backend/entrypoint.sh references config/ paths
- Configuration validation ensures required variables are present
- Logging shows which sources were used for debugging
- Backend starts successfully with new config structure
</success_criteria>
<output>
After completion, create `.planning/phases/07-config-consolidation/07-02-SUMMARY.md`
</output>

583
.planning/phases/07-config-consolidation/07-03-PLAN.md Normal file
View File

@@ -0,0 +1,583 @@
---
phase: 07-config-consolidation
plan: 03
type: execute
wave: 2
depends_on:
- 07-01
files_modified:
- scripts/deploy.py
- scripts/run_standalone.py
- scripts/install_service.py
- scripts/export_prod.py
autonomous: true
requirements:
- PHASE-7-PYTHON-SCRIPTS
- PHASE-7-YAML-PARSING
- PHASE-7-DEPLOYMENT
user_setup: []
must_haves:
truths:
- "Python deployment scripts (deploy.py, run_standalone.py, install_service.py, export_prod.py) exist and parse YAML config"
- "All scripts parse config/*.yaml files using PyYAML (per D-05)"
- "deploy.py handles Docker deployment with health checks"
- "run_standalone.py launches backend and frontend without Docker"
- "install_service.py installs systemd service with new config paths"
- "export_prod.py exports production data/config for backups"
- "All scripts are executable and tested"
artifacts:
- path: "scripts/deploy.py"
provides: "Docker deployment with YAML config parsing, pre-flight checks, health validation"
exports: ["main()"]
min_lines: 150
- path: "scripts/run_standalone.py"
provides: "Standalone launcher for backend (FastAPI) and frontend (Next.js) with YAML config"
exports: ["main()"]
min_lines: 120
- path: "scripts/install_service.py"
provides: "Systemd service installation with config paths"
exports: ["main()"]
min_lines: 100
- path: "scripts/export_prod.py"
provides: "Production export/backup script with YAML config support"
exports: ["main()"]
min_lines: 100
key_links:
- from: "scripts/deploy.py"
to: "config/docker.yaml"
via: "PyYAML parsing for container config"
pattern: "yaml\\.safe_load.*docker\\.yaml"
- from: "scripts/run_standalone.py"
to: "config/backend.yaml"
via: "Read port and path config"
pattern: "yaml\\.safe_load.*backend\\.yaml"
- from: "scripts/install_service.py"
to: "inventory.service.template"
via: "Service file generation"
pattern: "template|service"
---
<objective>
Convert bash deployment scripts (deploy.sh, run_standalone.sh, install_service.sh, export_prod.sh) to Python with YAML config parsing. Provide consistent, maintainable deployment tooling that understands the new config structure.
Purpose: Implement D-05 (Python scripts with YAML parsing) for modern deployment infrastructure.
Output: 4 Python scripts in scripts/ folder with full deployment functionality and YAML config support.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/07-config-consolidation/07-CONTEXT.md
@PROJECT_ARCHITECTURE.md
@DEPLOYMENT.md
@deploy.sh
@run_standalone.sh
@install_service.sh
@export_prod.sh
@config/backend.yaml.example
@config/docker.yaml.example
@docker-compose.yml
</context>
<tasks>
<task type="auto">
<name>Task 1: Create scripts/deploy.py (Docker deployment with YAML config)</name>
<files>
scripts/deploy.py
</files>
<read_first>
- deploy.sh (current bash implementation to port)
- docker-compose.yml (structure and environment variables)
- config/docker.yaml.example (schema)
- config/backend.yaml.example (config structure)
- DEPLOYMENT.md (deployment procedure documentation)
</read_first>
<action>
Create scripts/deploy.py to replace deploy.sh with Python implementation (per D-05).
**Key features:**
1. **Script signature and usage:**
```bash
python3 scripts/deploy.py [production|staging|development] [--rebuild]
```
2. **Core functionality:**
- Pre-flight checks: Docker, Docker Compose, docker-compose.yml, config files
- Load config from config/docker.yaml and config/network.yaml
- Port availability checks (from network.yaml: backend_port, frontend_port, etc.)
- Environment file validation (config/backend.yaml exists and has required values)
- Docker Compose up with appropriate flags (rebuild if --rebuild)
- Health check polling (curl to /health endpoints)
- Deployment report (services running, ports, access URLs)
3. **Config file parsing:**
- Use PyYAML to load config/docker.yaml (for container resource limits, image names)
- Use PyYAML to load config/network.yaml (for port numbers and SSL settings)
- Use PyYAML to load config/backend.yaml (to validate required values)
- Fallback to sensible defaults if config files missing (but log warnings)
4. **Pre-flight checks (Step 1-5):**
- [ ] docker command available
- [ ] docker-compose command available
- [ ] docker-compose.yml exists
- [ ] config/backend.yaml exists (with helpful error if missing)
- [ ] config/network.yaml exists (with helpful error if missing)
5. **Port availability check (Step 6):**
- Read backend_port, frontend_port, backend_ssl_port, frontend_ssl_port from network.yaml
- Use netstat or ss to check if ports are in use
- Error if ports occupied, suggest alternatives
6. **Environment validation (Step 7):**
- Check config/backend.yaml for required values: JWT_SECRET_KEY, primary_ai_provider
- Warn if API keys are placeholders
- Proceed with warning (not error) for optional values
7. **Docker Compose deployment (Step 8-9):**
- Run `docker-compose up -d` (or with --build if --rebuild flag)
- Capture and display output with color codes
- Catch errors and provide helpful debugging steps
8. **Health checks (Step 10-11):**
- Poll backend health: `curl http://localhost:{backend_port}/health` (retry logic)
- Poll frontend health: `curl http://localhost:{frontend_port}/` (retry logic)
- Wait up to 2 minutes for services to become healthy
- Display health status to user
9. **Deployment report (Step 12):**
- Display service status: `docker-compose ps`
- Display access URLs:
- Frontend: http://localhost:{frontend_port}
- Backend API: http://localhost:{backend_port}/docs
- HTTPS: https://localhost:{frontend_ssl_port} (if SSL enabled in network.yaml)
- Display next steps (logs, troubleshooting, etc.)
10. **Error handling:**
- Descriptive error messages with suggested fixes
- Log all actions and results to stdout/stderr
- Use color output (GREEN for success, RED for errors, YELLOW for warnings, BLUE for info)
- Exit codes: 0 for success, 1 for fatal error
11. **Logging:**
- Use Python logging module (not print)
- Log level: INFO by default, DEBUG if --verbose flag
- Each step logged: "Step N/M: Description..."
- Results logged at end: "Deployment complete, services healthy"
12. **Required libraries:**
- sys, os, subprocess, time, socket (built-in)
- yaml (PyYAML)
- argparse (for CLI args)
- logging (for logging)
- No external deployment libraries (keep it simple)
13. **Make executable:** `chmod +x scripts/deploy.py` and include shebang: `#!/usr/bin/env python3`
</action>
<verify>
- `test -f scripts/deploy.py && head -1 scripts/deploy.py | grep -q "python3"` (shebang present)
- `test -x scripts/deploy.py` (executable)
- `python3 -m py_compile scripts/deploy.py` (valid Python syntax)
- `python3 scripts/deploy.py --help | grep -q "deployment"` (help works)
- `grep -q "import yaml" scripts/deploy.py` (PyYAML imported)
- `grep -q "config/docker.yaml\|config/network.yaml" scripts/deploy.py` (loads config files)
- `grep -q "docker-compose" scripts/deploy.py` (calls docker-compose)
- `grep -q "curl.*health" scripts/deploy.py` (health checks present)
</verify>
<done>
scripts/deploy.py created with Docker deployment, YAML config parsing, health checks, and error handling.
</done>
</task>
<task type="auto">
<name>Task 2: Create scripts/run_standalone.py (Standalone launcher with YAML config)</name>
<files>
scripts/run_standalone.py
</files>
<read_first>
- run_standalone.sh (current bash implementation to port)
- config/backend.yaml.example (schema)
- config/frontend.yaml.example (schema)
- backend/main.py (backend entry point)
- frontend package.json or next.config.js (frontend startup)
</read_first>
<action>
Create scripts/run_standalone.py to replace run_standalone.sh with Python implementation (per D-05).
**Key features:**
1. **Script signature:**
```bash
python3 scripts/run_standalone.py [--backend-only|--frontend-only]
```
2. **Core functionality:**
- Load config from config/backend.yaml and config/frontend.yaml
- Start FastAPI backend (uvicorn)
- Start Next.js frontend (npm run dev or node server.js)
- Display console output from both processes
- Handle shutdown gracefully (SIGTERM/SIGINT kills both services)
- Display health status and access URLs
3. **Config file parsing:**
- Load config/backend.yaml to get: backend_port, data_dir, logs_dir, log_level
- Load config/frontend.yaml to get: frontend_port, backend_url
- Use defaults if config files missing (with warnings)
4. **Backend startup (--backend-only or default):**
- Command: `uvicorn backend.main:app --host 0.0.0.0 --port {backend_port} --reload`
- Set environment: DATA_DIR, LOGS_DIR, LOG_LEVEL (from config)
- Capture output and display with [BACKEND] prefix
- Wait for backend to log "Uvicorn running on..." or similar
- Verify backend is listening on backend_port
5. **Frontend startup (--frontend-only or default):**
- Command: `npm run dev` (if in development) or `node server.js` (if built)
- Set environment: NEXT_PUBLIC_API_URL (from config:frontend:backend_url)
- Capture output and display with [FRONTEND] prefix
- Wait for frontend to log "ready - started server on..." or similar
- Verify frontend is listening on frontend_port
6. **Process management:**
- Use subprocess.Popen with shell=False (for security)
- Manage both processes in list/tuple
- Handle SIGTERM/SIGINT (Ctrl+C) to kill both processes
- Display "Shutting down..." and wait for clean shutdown
- Exit with code 0 if both shut down cleanly
7. **Health monitoring:**
- Periodically check if processes are alive (poll returncode)
- If one process dies, log error and optionally shutdown other (per config flag)
- Display uptime and status every 30 seconds
8. **Logging and output:**
- Use Python logging module
- Log each process with [BACKEND] / [FRONTEND] prefix
- Merge stdout/stderr from both processes to terminal
- Show final status: "Backend running on http://localhost:{backend_port}, Frontend on http://localhost:{frontend_port}"
9. **Error handling:**
- If uvicorn not installed, error and suggest: `pip install uvicorn`
- If npm not found, error and suggest: install Node.js
- If ports already in use, error with port number
- If config files missing, log warnings but use defaults
10. **Required libraries:**
- sys, os, subprocess, signal, time (built-in)
- yaml (PyYAML)
- argparse (for CLI args --backend-only, --frontend-only)
- logging (for logging)
11. **Make executable:** `chmod +x scripts/run_standalone.py` with shebang: `#!/usr/bin/env python3`
</action>
<verify>
- `test -f scripts/run_standalone.py && head -1 scripts/run_standalone.py | grep -q "python3"` (shebang present)
- `test -x scripts/run_standalone.py` (executable)
- `python3 -m py_compile scripts/run_standalone.py` (valid Python syntax)
- `grep -q "import yaml" scripts/run_standalone.py` (PyYAML imported)
- `grep -q "config/backend.yaml\|config/frontend.yaml" scripts/run_standalone.py` (loads config)
- `grep -q "uvicorn\|subprocess.Popen" scripts/run_standalone.py` (backend startup present)
- `grep -q "npm\|node server" scripts/run_standalone.py` (frontend startup present)
- `grep -q "signal.signal\|SIGTERM" scripts/run_standalone.py` (signal handling present)
</verify>
<done>
scripts/run_standalone.py created with backend/frontend startup, YAML config parsing, process management, and graceful shutdown.
</done>
</task>
<task type="auto">
<name>Task 3: Create scripts/install_service.py (Systemd service installation)</name>
<files>
scripts/install_service.py
</files>
<read_first>
- install_service.sh (current bash implementation to port)
- inventory.service.template (systemd service template)
- config/backend.yaml.example (to understand config structure)
- config/network.yaml.example (for port information)
</read_first>
<action>
Create scripts/install_service.py to replace install_service.sh with Python implementation (per D-05).
**Key features:**
1. **Script signature:**
```bash
sudo python3 scripts/install_service.py [--user=service_user] [--port=port]
```
2. **Core functionality:**
- Read inventory.service.template (or create template inline)
- Load config from config/backend.yaml (for paths, ports)
- Generate systemd service file with correct paths and user/group
- Install service file to /etc/systemd/system/ainventory.service
- Enable service (systemctl enable)
- Display installation summary and next steps
3. **Config file parsing:**
- Load config/backend.yaml to get: data_dir, logs_dir
- Load config/network.yaml to get: backend_port (for documentation)
- Use defaults if missing
4. **Service file generation:**
- Read inventory.service.template
- Replace placeholders:
- {PROJECT_DIR}: current working directory (project root)
- {SERVICE_USER}: service user (default: www-data, configurable via --user)
- {BACKEND_PORT}: from config/network.yaml
- {DATA_DIR}: from config/backend.yaml
- {LOGS_DIR}: from config/backend.yaml
- Template should:
- Type=simple
- ExecStart=/usr/bin/python3 {PROJECT_DIR}/scripts/run_standalone.py --backend-only
- WorkingDirectory={PROJECT_DIR}
- User={SERVICE_USER}
- Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
- Restart=on-failure
- RestartSec=10
5. **Permission checks:**
- Require sudo/root: `if os.getuid() != 0: error("Must run with sudo")`
- Check project directory is readable
- Check data_dir and logs_dir exist or can be created
6. **Service file installation:**
- Write service file to /etc/systemd/system/ainventory.service
- Set permissions: 644 (readable, not writable by non-root)
- Run `systemctl daemon-reload`
- Run `systemctl enable ainventory.service`
- Optionally start service: `systemctl start ainventory.service`
7. **Status display:**
- Show service file location
- Show service user and group
- Show project directory
- Show next steps: `systemctl start ainventory`, `systemctl status ainventory`
- Show logs: `journalctl -u ainventory -f`
8. **Error handling:**
- Check if service already installed (offer --force to overwrite)
- Check if user exists (suggest: `useradd -r -s /bin/false {user}`)
- Check if directories are writable
- Descriptive errors with suggested fixes
9. **Required libraries:**
- sys, os, subprocess, pwd, grp (built-in)
- yaml (PyYAML)
- argparse (for CLI args)
- logging (for logging)
10. **Make executable:** `chmod +x scripts/install_service.py` with shebang: `#!/usr/bin/env python3`
</action>
<verify>
- `test -f scripts/install_service.py && head -1 scripts/install_service.py | grep -q "python3"` (shebang present)
- `test -x scripts/install_service.py` (executable)
- `python3 -m py_compile scripts/install_service.py` (valid Python syntax)
- `grep -q "import yaml" scripts/install_service.py` (PyYAML imported)
- `grep -q "config/backend.yaml\|config/network.yaml" scripts/install_service.py` (loads config)
- `grep -q "/etc/systemd/system\|systemctl" scripts/install_service.py` (systemd integration present)
- `grep -q "os.getuid\|sudo" scripts/install_service.py` (permission check present)
</verify>
<done>
scripts/install_service.py created with systemd service generation, config parsing, and installation logic.
</done>
</task>
<task type="auto">
<name>Task 4: Create scripts/export_prod.py (Production export/backup)</name>
<files>
scripts/export_prod.py
</files>
<read_first>
- export_prod.sh (current bash implementation to port)
- config/backend.yaml.example (for data_dir)
- DEPLOYMENT.md (backup procedures)
</read_first>
<action>
Create scripts/export_prod.py to replace export_prod.sh with Python implementation (per D-05).
**Key features:**
1. **Script signature:**
```bash
python3 scripts/export_prod.py [--output=/path/to/backup.tar.gz] [--include-logs]
```
2. **Core functionality:**
- Load config from config/backend.yaml (to find data_dir, logs_dir)
- Create tar.gz archive of production data
- Include database file(s), config files (no secrets), and optionally logs
- Output to specified location or default: backups/{timestamp}.tar.gz
- Display archive size and location
3. **Config file parsing:**
- Load config/backend.yaml to get: data_dir, logs_dir
- Use defaults if missing: data_dir=./data, logs_dir=./logs
4. **Archive creation:**
- Include: {data_dir}/* (all application data, database, etc.)
- Include: config/*.yaml.example (config templates)
- Include: config/backend.yaml, config/frontend.yaml, config/network.yaml (actual configs, no secrets)
- Include: config/secrets.yaml.example (secrets template only, NOT actual secrets.yaml)
- Include: logs/* (optional, if --include-logs flag)
- Exclude: config/secrets.yaml (never backup actual secrets)
- Exclude: node_modules/, __pycache__/, .git/, .venv/
- Exclude: temporary files, cache
5. **Archive naming:**
- Default: backups/ainventory_{timestamp}.tar.gz
- Timestamp format: YYYY-MM-DD_HH-MM-SS
- Custom path via --output flag
6. **Backup directory:**
- Create backups/ directory if not exists
- Set directory permissions: 750 (rwxr-x---)
7. **Verification:**
- Verify tar.gz was created successfully
- Display archive size: X.XX MB
- Display archive contents summary: "Includes database, data, and config (secrets excluded)"
8. **Error handling:**
- If data_dir doesn't exist, error and suggest creating it
- If no write permission to backups/, error and suggest location
- If tar command fails, show error and suggest troubleshooting
9. **Output example:**
```
[INFO] Loading config from config/backend.yaml
[INFO] Data directory: ./data
[INFO] Creating backup...
[INFO] Archive created: backups/ainventory_2026-04-23_14-30-45.tar.gz
[INFO] Archive size: 125.43 MB
[INFO] Contents: database, data, config (secrets excluded)
[INFO] Backup complete!
```
10. **Required libraries:**
- sys, os, subprocess, datetime, tarfile (built-in)
- yaml (PyYAML)
- argparse (for CLI args)
- logging (for logging)
11. **Make executable:** `chmod +x scripts/export_prod.py` with shebang: `#!/usr/bin/env python3`
</action>
<verify>
- `test -f scripts/export_prod.py && head -1 scripts/export_prod.py | grep -q "python3"` (shebang present)
- `test -x scripts/export_prod.py` (executable)
- `python3 -m py_compile scripts/export_prod.py` (valid Python syntax)
- `grep -q "import yaml" scripts/export_prod.py` (PyYAML imported)
- `grep -q "config/backend.yaml" scripts/export_prod.py` (loads config)
- `grep -q "tarfile\|tar.gz" scripts/export_prod.py` (tar archive creation present)
- `grep -q "secrets.yaml" scripts/export_prod.py | grep -q "exclude"` (secrets excluded from backup)
</verify>
<done>
scripts/export_prod.py created with production data export, YAML config parsing, archive creation, and secrets exclusion.
</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| User input → Script | Script arguments and config files must be validated |
| Script → System | Scripts execute system commands (subprocess) — must escape/quote properly |
| Script → Network | Health checks make HTTP requests (must handle timeouts) |
| Script → Filesystem | Scripts read/write files (must respect permissions) |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-07-09 | Injection | deploy.py subprocess | mitigate | Use subprocess with shell=False and list args (not f-strings). Example: `subprocess.run(["docker-compose", "up", "-d"], ...)` not `subprocess.run(f"docker-compose up -d", shell=True)`. |
| T-07-10 | Elevation of Privilege | install_service.py sudo | mitigate | Check `os.getuid() != 0` at start. Require sudo for systemd operations only. Log all systemctl calls. |
| T-07-11 | Information Disclosure | export_prod.py backup | mitigate | Exclude config/secrets.yaml explicitly in tarfile. Log what is excluded. Verify file permissions (backups/ dir 750). |
| T-07-12 | Denial of Service | Health checks timeout | mitigate | Set socket timeout to 10 seconds. Limit retry attempts to 12 (2 minutes total). Log timeout errors. |
</threat_model>
<verification>
**Phase 7, Plan 3 Verification Checklist:**
1. **scripts/deploy.py**
- [ ] File exists and is executable
- [ ] Shebang present: `#!/usr/bin/env python3`
- [ ] Loads config/docker.yaml and config/network.yaml
- [ ] Pre-flight checks for Docker, Docker Compose, config files
- [ ] Port availability checks implemented
- [ ] Health checks poll backend and frontend endpoints
- [ ] Color output for info/warning/success/error
- [ ] Displays deployment summary and access URLs
- [ ] Valid Python syntax
2. **scripts/run_standalone.py**
- [ ] File exists and is executable
- [ ] Shebang present: `#!/usr/bin/env python3`
- [ ] Loads config/backend.yaml and config/frontend.yaml
- [ ] Launches uvicorn for backend with correct port and settings
- [ ] Launches frontend (npm dev or node server.js) with correct port
- [ ] Signal handling (SIGTERM/SIGINT) for clean shutdown
- [ ] Process monitoring and output display with prefixes
- [ ] Valid Python syntax
3. **scripts/install_service.py**
- [ ] File exists and is executable
- [ ] Shebang present: `#!/usr/bin/env python3`
- [ ] Checks for sudo/root permission
- [ ] Loads config/backend.yaml and config/network.yaml
- [ ] Generates systemd service file from template
- [ ] Replaces placeholders: {PROJECT_DIR}, {SERVICE_USER}, {BACKEND_PORT}, etc.
- [ ] Installs to /etc/systemd/system/ with correct permissions
- [ ] Runs systemctl daemon-reload and enable
- [ ] Valid Python syntax
4. **scripts/export_prod.py**
- [ ] File exists and is executable
- [ ] Shebang present: `#!/usr/bin/env python3`
- [ ] Loads config/backend.yaml to find data_dir
- [ ] Creates tar.gz archive with data and config files
- [ ] Excludes config/secrets.yaml (actual secrets, not example)
- [ ] Includes config/*.yaml.example files
- [ ] Output to backups/{timestamp}.tar.gz or custom path
- [ ] Displays archive size and summary
- [ ] Valid Python syntax
5. **Security (subprocess, permissions, file ops)**
- [ ] All subprocess calls use shell=False with list args
- [ ] No f-strings in shell commands
- [ ] File operations respect umask/permissions
- [ ] No hardcoded credentials in scripts
6. **Integration**
- [ ] Each script loads YAML config files correctly
- [ ] Scripts reference new config/ structure (not inventory.env)
- [ ] Error messages are helpful and actionable
</verification>
<success_criteria>
- 4 Python scripts created (deploy.py, run_standalone.py, install_service.py, export_prod.py)
- All scripts use PyYAML to parse config files
- All scripts are executable with proper shebangs
- deploy.py handles Docker deployment with health checks
- run_standalone.py launches backend and frontend without Docker
- install_service.py creates systemd service with new config paths
- export_prod.py exports production data excluding secrets
- All subprocess calls use shell=False (secure)
- Error handling and logging present in all scripts
- Scripts integrate with new config/ structure (D-05, D-06, D-07)
</success_criteria>
<output>
After completion, create `.planning/phases/07-config-consolidation/07-03-SUMMARY.md`
</output>

642
.planning/phases/07-config-consolidation/07-04-PLAN.md Normal file
View File

@@ -0,0 +1,642 @@
---
phase: 07-config-consolidation
plan: 04
type: execute
wave: 3
depends_on:
- 07-01
- 07-02
- 07-03
files_modified:
- docker-compose.yml
- backend/Dockerfile
- backend/entrypoint.sh
- .gitignore
- DEPLOYMENT.md
- README.md
autonomous: true
requirements:
- PHASE-7-DOCKER-UPDATE
- PHASE-7-DOCUMENTATION
- PHASE-7-GITIGNORE
user_setup: []
must_haves:
truths:
- "docker-compose.yml updated to reference config/ volume and remove inventory.env env_file"
- "backend/Dockerfile and entrypoint updated for new config paths"
- ".gitignore properly configured to track examples, ignore actual configs and secrets"
- "DEPLOYMENT.md updated with YAML config structure, new Python scripts, setup instructions"
- "README.md updated with config setup and configuration management instructions"
- "Docker deployment works with new config structure (tested with docker-compose up)"
- "All documentation references config/ as single source of truth"
artifacts:
- path: "docker-compose.yml"
provides: "Docker Compose with config/ volume mount, no inventory.env env_file reference"
pattern: "\\./config:/app/config|!inventory.env"
min_lines: 120
- path: "backend/Dockerfile"
provides: "Backend container image with new config paths"
pattern: "config/|/app/config"
- path: "backend/entrypoint.sh"
provides: "Docker entrypoint with config/ reference"
pattern: "config/|/app/config"
- path: "DEPLOYMENT.md"
provides: "Updated deployment guide with YAML config structure and Python scripts"
min_lines: 150
- path: "README.md"
provides: "Updated README with config setup and onboarding"
min_lines: 100
- path: ".gitignore"
provides: ".gitignore with rules for config/ folder (track examples, ignore secrets)"
pattern: "config/.*\\.yaml"
key_links:
- from: "docker-compose.yml"
to: "config/"
via: "volume mount"
pattern: "\\./config:/app/config"
- from: "DEPLOYMENT.md"
to: "config/README.md"
provides: "Cross-reference to config documentation"
pattern: "config/README.md|config/"
- from: "README.md"
to: "DEPLOYMENT.md"
provides: "Cross-reference to deployment guide"
pattern: "DEPLOYMENT.md|config/"
---
<objective>
Update Docker Compose, Dockerfile, documentation, and .gitignore to integrate the new config/ structure. Remove references to inventory.env from deployment infrastructure and update all deployment documentation.
Purpose: Complete D-07 (Docker & Compose update), D-08 (documentation), and D-04 (deprecation) for cohesive deployment experience.
Output: Updated docker-compose.yml, Dockerfile, entrypoint.sh, DEPLOYMENT.md, README.md, and .gitignore.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/07-config-consolidation/07-CONTEXT.md
@PROJECT_ARCHITECTURE.md
@docker-compose.yml
@backend/Dockerfile
@backend/entrypoint.sh
@DEPLOYMENT.md
@README.md
@.gitignore
@config/README.md
</context>
<tasks>
<task type="auto">
<name>Task 1: Update docker-compose.yml to reference config/ and remove inventory.env env_file</name>
<files>
docker-compose.yml
</files>
<read_first>
- docker-compose.yml (current file)
- backend/entrypoint.sh (to understand how config is used in containers)
- config/backend.yaml.example (to understand what config is needed)
- config/docker.yaml.example (Docker-specific config)
</read_first>
<action>
Update docker-compose.yml to integrate new config/ structure (per D-07):
1. **Remove inventory.env env_file references:**
- Delete or comment out `env_file: - inventory.env` from backend service (currently line 14)
- Delete or comment out `env_file: - inventory.env` from frontend service (currently line 51)
- Keep proxy service as is (may not need env_file)
2. **Add config/ volume mount to backend service:**
- Keep existing volume mounts
- Add: `- ./config:/app/config:ro` (read-only, config should not be modified in container)
- Update volumes section to reflect new mount
3. **Add config/ volume mount to frontend service (if frontend needs config):**
- Add: `- ./config:/app/config:ro` if frontend needs to read config files
- Or skip if frontend doesn't read YAML config directly
4. **Update environment variables:**
- Keep all existing environment variables (Docker overrides are still valid per D-06)
- Ensure JWT_SECRET_KEY is still set with warning: `# CHANGE THIS IN PRODUCTION!`
- Add comment: "Environment variables override config/backend.yaml per D-06 load order"
- Ensure DATA_DIR and LOGS_DIR are set to persist volume locations
5. **Add proxy service config/ mount (if proxy reads config):**
- Check if Caddyfile uses any dynamic config
- If not, no change needed
- If yes, add: `- ./config:/app/config:ro`
6. **Verify volume definitions:**
- Named volumes (backend_data, backend_logs, frontend_logs, caddy_data, caddy_config) remain unchanged
- Config mount is bind mount (./config), not named volume
7. **Add comments explaining the change:**
- Add section comment before volumes: "# [D-07] New config/ structure — YAML config mounted read-only"
- Add comment on env_file removal: "# [D-04] inventory.env deprecated — config now in config/ folder"
- Reference Phase 7 decisions
8. **Maintain backward compatibility during transition:**
- Don't delete old env_file line yet (can exist but be ignored by modern docker-compose)
- Or clearly comment it out with deprecation notice
Example section after update:
```yaml
backend:
...
# [D-04] inventory.env deprecated — see config/ folder instead
# env_file: - inventory.env
volumes:
- backend_data:/app/data
- backend_logs:/app/logs
# [D-07] New config/ structure mounted read-only
- ./config:/app/config:ro
- ./scripts:/app/scripts:ro
...
```
</action>
<verify>
- `grep -n "env_file" docker-compose.yml | head` (check if inventory.env env_file is removed/commented)
- `grep -q "\\./config:/app/config:ro" docker-compose.yml` (config volume mount present)
- `docker-compose config 2>&1 | grep -q "config" || echo "valid yaml"` (valid docker-compose syntax)
- `grep -q "D-07\|D-04" docker-compose.yml || echo "pass"` (comments reference phase decisions, optional)
</verify>
<done>
docker-compose.yml updated with config/ volume mount, inventory.env env_file removed, comments documenting changes.
</done>
</task>
<task type="auto">
<name>Task 2: Update backend/Dockerfile for new config paths</name>
<files>
backend/Dockerfile
</files>
<read_first>
- backend/Dockerfile (current file)
- docker-compose.yml (just updated)
- backend/entrypoint.sh (how config is used at runtime)
</read_first>
<action>
Update backend/Dockerfile to document/support new config/ paths (per D-07):
1. **Add comments explaining config/ structure:**
- Add comment at top: "# [D-07] Backend container - config/ folder mounted at /app/config (read-only)"
- Add comment before WORKDIR: "# Config is expected in /app/config (mounted from host)"
2. **Ensure volume mount points exist:**
- Config is mounted at runtime by docker-compose, not created in Dockerfile
- No changes needed to RUN commands for config directory
- (Already handled by docker-compose volume mount)
3. **Update any hardcoded paths referencing inventory.env:**
- Search for "inventory.env" in Dockerfile
- Replace with reference to config/ or remove if no longer needed
- Example: If old CMD references inventory.env, update to reference config/
4. **Update ENTRYPOINT or CMD if needed:**
- Ensure entrypoint.sh (or equivalent) references config/ paths
- Add environment documentation: "# Config sources: /app/config/backend.yaml, /app/config/secrets.yaml, environment variables"
5. **Add healthcheck if not present:**
- Verify backend has healthcheck (curl to /health endpoint)
- Should already be in docker-compose.yml, but double-check Dockerfile
6. **Document environment variables:**
- Add comment: "# Environment variables override YAML config per D-06"
- List: DATA_DIR, LOGS_DIR, LOG_LEVEL (these come from env and/or config)
Example after update:
```dockerfile
# [D-07] Backend container - config/ folder mounted at /app/config (read-only)
# Config sources: /app/config/backend.yaml, /app/config/secrets.yaml, environment variables
# Environment variables override YAML config per D-06 load order
FROM python:3.12-slim
WORKDIR /app
# Copy code, requirements, and startup scripts
COPY backend/ ./backend/
COPY scripts/ ./scripts/
COPY requirements.txt .
# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt
# Config is mounted at /app/config by docker-compose
# No need to COPY config/ here (it's mounted read-only)
ENTRYPOINT ["python", "backend/main.py"]
```
</action>
<verify>
- `grep -q "D-07\|config/" backend/Dockerfile || echo "pass"` (comments reference config, optional)
- `grep -q "inventory.env" backend/Dockerfile` should return empty (no old inventory.env refs)
- `docker build -f backend/Dockerfile .` (valid Dockerfile syntax — may not succeed without full context, but no syntax errors)
</verify>
<done>
backend/Dockerfile updated with comments documenting config/ structure, no inventory.env references.
</done>
</task>
<task type="auto">
<name>Task 3: Update backend/entrypoint.sh for new config paths</name>
<files>
backend/entrypoint.sh
</files>
<read_first>
- backend/entrypoint.sh (current file)
- backend/config_loader.py (updated in Plan 2, to understand config loading)
- config/README.md (documentation on config structure)
</read_first>
<action>
Update backend/entrypoint.sh to reference and support new config/ structure (per D-07):
1. **Remove inventory.env sourcing:**
- Delete any lines that source or check for inventory.env
- Delete any EXPORT statements that copy inventory.env values to environment
2. **Add config/ path documentation:**
- Add comment at top: "# [D-07] Backend entrypoint - loads config from /app/config/ (YAML format)"
- Add comment: "# Config sources: /app/config/backend.yaml, /app/config/secrets.yaml, environment variables"
3. **Add environment variable override documentation:**
- Add comment: "# [D-06] Environment variables override YAML config — set below takes precedence"
- List typical overrides: JWT_SECRET_KEY, PRIMARY_AI_PROVIDER, LOG_LEVEL, etc.
4. **Ensure config validation:**
- Add check: if [ ! -f "/app/config/backend.yaml" ]; then log error and instructions
- Add comment: "# Config validation handled by Python config_loader.py"
5. **Set working directory:**
- Ensure WORKDIR is set to /app (should be done in Dockerfile, but double-check)
6. **Exec main process:**
- Ensure entrypoint uses `exec` to replace shell: `exec python backend/main.py`
- This ensures signals (SIGTERM) are properly handled by Python process
Example after update:
```bash
#!/bin/bash
# [D-07] Backend entrypoint - loads config from /app/config/ (YAML format)
# Config sources: /app/config/backend.yaml, /app/config/secrets.yaml, environment variables
# [D-06] Environment variables override YAML config (below takes precedence)
set -euo pipefail
cd /app
# Verify config is accessible
if [ ! -f "/app/config/backend.yaml" ]; then
echo "[ERROR] /app/config/backend.yaml not found!"
echo "[ERROR] Config must be mounted from host at /app/config/"
echo "[ERROR] See config/README.md for setup instructions"
exit 1
fi
# Environment variables below override YAML config
# (docker run -e JWT_SECRET_KEY="..." or docker-compose environment)
# Start backend (signals properly handled with exec)
exec python -m uvicorn backend.main:app --host 0.0.0.0 --port 8000
```
7. **Keep it minimal:**
- Entrypoint should be simple (most logic in Python config_loader.py)
- Just verify config exists and start the app
</action>
<verify>
- `grep -q "D-07\|D-06\|config/" backend/entrypoint.sh` (references config structure, optional)
- `grep -q "inventory.env" backend/entrypoint.sh` should return empty (no old config)
- `bash -n backend/entrypoint.sh` (valid bash syntax)
- `head -1 backend/entrypoint.sh | grep -q "bash"` (shebang present)
</verify>
<done>
backend/entrypoint.sh updated to reference config/ paths, remove inventory.env, document env var overrides.
</done>
</task>
<task type="auto">
<name>Task 4: Update .gitignore to track config examples and ignore actual configs/secrets</name>
<files>
.gitignore
</files>
<read_first>
- .gitignore (current file)
- config/backend.yaml.example (created in Plan 1)
- config/secrets.yaml.example (created in Plan 1)
</read_first>
<action>
Update .gitignore to properly handle config/ folder (per D-03, D-08):
1. **Add config/ rules:**
```
# [D-08] Config folder — track examples, ignore actual configs and secrets
config/*.yaml
!config/*.yaml.example
config/secrets.yaml
!config/secrets.yaml.example
```
2. **Rationale:**
- `config/*.yaml` — Ignore all YAML files (actual configs with real secrets)
- `!config/*.yaml.example` — Except examples (these are tracked for schema/documentation)
- `config/secrets.yaml` — Explicitly ignore secrets file (redundant but clear)
- `!config/secrets.yaml.example` — Except example (for developer setup guidance)
3. **Clean up old rules:**
- Remove any existing `inventory.env` entries from .gitignore (deprecated)
- Or update to comment them as deprecated: `# inventory.env # [D-04] Deprecated - use config/backend.yaml`
4. **Add comment at top of config section:**
- Add comment: "# [D-04] inventory.env deprecated — see config/ folder instead"
- Add comment: "# [D-08] Config structure: examples tracked, actual configs ignored"
5. **Example .gitignore config section:**
```
# [D-04] inventory.env deprecated — see config/ folder instead
# [D-08] Config structure: examples tracked (schema), actual configs ignored (secrets)
config/*.yaml
!config/*.yaml.example
config/secrets.yaml
!config/secrets.yaml.example
```
6. **Verify git status:**
- After update, git status should show:
- config/*.yaml.example as "new file" or tracked
- config/*.yaml as ignored
- config/secrets.yaml as ignored
</action>
<verify>
- `grep -q "config/\\*\\.yaml" .gitignore` (config rule present)
- `grep -q "!config/\\*\\.yaml\\.example" .gitignore` (exception for examples present)
- `grep -q "config/secrets\\.yaml" .gitignore` (secrets ignored)
- `grep -q "!config/secrets\\.yaml\\.example" .gitignore` (example tracked)
</verify>
<done>
.gitignore updated with config/ rules to track examples and ignore actual configs/secrets.
</done>
</task>
<task type="auto">
<name>Task 5: Update DEPLOYMENT.md with YAML config structure and new Python scripts</name>
<files>
DEPLOYMENT.md
</files>
<read_first>
- DEPLOYMENT.md (current file)
- config/README.md (created in Plan 1)
- scripts/deploy.py, scripts/run_standalone.py (created in Plan 3)
</read_first>
<action>
Update DEPLOYMENT.md to document new YAML config structure and Python deployment scripts (per D-08):
1. **Add section: Configuration (Before Quick Start)**
- Explain config/ as single source of truth
- List config files: backend.yaml, frontend.yaml, network.yaml, docker.yaml, secrets.yaml
- Reference config/README.md for detailed setup
- Explain examples and how to create actual config from examples
2. **Update Quick Start section:**
- Step 1: Clone and cd
- Step 2: Copy config examples to actual files: `cp config/*.yaml.example {without .example}`
- Step 3: Edit config files (backend.yaml, network.yaml, secrets.yaml) with your values
- Step 4: Choose deployment mode (Docker or Standalone)
3. **Update Docker Deployment section:**
- Replace old deploy.sh instructions with new deploy.py
- Usage: `python3 scripts/deploy.py [production|staging|development] [--rebuild]`
- Script handles: pre-flight checks, port validation, config loading, health checks
- Output: service status, access URLs
4. **Update Standalone Deployment section:**
- Replace old run_standalone.sh with new run_standalone.py
- Usage: `python3 scripts/run_standalone.py [--backend-only|--frontend-only]`
- Script handles: config loading, backend startup, frontend startup, signal handling
5. **Update Configuration Reference section:**
- Refer to config/README.md for complete reference
- List key files: config/backend.yaml, config/frontend.yaml, config/network.yaml, config/docker.yaml, config/secrets.yaml
- Explain environment variable overrides (D-06)
6. **Add Systemd Service Installation section:**
- Usage: `sudo python3 scripts/install_service.py [--user=www-data]`
- Explain what service does: runs standalone backend + frontend
- Show how to manage: `systemctl start|stop|status ainventory`
- Show logs: `journalctl -u ainventory -f`
7. **Add Backup & Export section:**
- Usage: `python3 scripts/export_prod.py [--output=/path/to/backup.tar.gz] [--include-logs]`
- Explains what is included: data, config, config templates
- Explains what is excluded: actual secrets (security), logs (optional)
8. **Add Security section:**
- Mention secrets.yaml is git-ignored
- Explain how to set up secrets (copy from example, fill in values)
- Warn about JWT_SECRET_KEY in docker-compose.yml (must change for production)
9. **Add Troubleshooting section:**
- Common issues: missing config files, invalid YAML syntax, port conflicts
- Debug steps: check config syntax, verify file permissions, run health checks
- Reference config/README.md for setup help
10. **Add Migration section (from old inventory.env):**
- Explain Phase 7 transition from inventory.env to config/ structure
- Provide migration script or manual steps
- Clear instructions: which old values map to which config files
Structure should be roughly:
- 1. Overview (unchanged)
- 2. Prerequisites (unchanged)
- **3. Configuration** (NEW)
- 4. Quick Start (UPDATED)
- 5. Deployment Modes (Docker, Standalone) (UPDATED to use Python scripts)
- 6. Systemd Service (UPDATED with Python script)
- 7. Backup & Export (UPDATED with Python script)
- 8. Operations & Health Monitoring (UPDATED with new paths)
- 9. Security (NEW or UPDATED)
- 10. Troubleshooting (UPDATED)
- 11. Migration from inventory.env (NEW)
</action>
<verify>
- `grep -q "config/.*\\.yaml" DEPLOYMENT.md` (references YAML config files)
- `grep -q "scripts/deploy\\.py\|scripts/run_standalone\\.py" DEPLOYMENT.md` (references Python scripts)
- `grep -q "config/README\\.md" DEPLOYMENT.md` (cross-references config documentation)
- `grep -q "environment.*override\|D-06" DEPLOYMENT.md || echo "pass"` (explains env var overrides, optional)
- File should have 150+ lines: `wc -l DEPLOYMENT.md | awk '$1 >= 150 {print "pass"}'`
</verify>
<done>
DEPLOYMENT.md updated with YAML config structure, Python script usage, systemd service, backup procedures, and troubleshooting.
</done>
</task>
<task type="auto">
<name>Task 6: Update README.md with config setup and configuration management instructions</name>
<files>
README.md
</files>
<read_first>
- README.md (current file)
- DEPLOYMENT.md (just updated)
- config/README.md (created in Plan 1)
</read_first>
<action>
Update README.md to include configuration management and quick setup instructions (per D-08):
1. **Add Configuration section (after Quick Start or before Deployment):**
- Brief explanation: config/ folder is single source of truth
- Quick steps: cp config/*.example to remove .example, edit with your values
- Reference config/README.md for detailed setup
- Reference DEPLOYMENT.md for deployment options
2. **Update Quick Start section (if exists):**
- Add configuration step before deployment
- Example:
```
# 1. Clone and setup
git clone ... && cd tfm-inventory
# 2. Configure application
cp config/*.yaml.example config/$(basename {} .example) # or similar
nano config/backend.yaml # Edit with your values
cp config/secrets.yaml.example config/secrets.yaml
nano config/secrets.yaml # Fill in API keys and secrets
# 3. Deploy (choose one)
python3 scripts/deploy.py production # Docker
# OR
python3 scripts/run_standalone.py # Standalone
```
3. **Add note about .gitignore:**
- Config examples are tracked (for schema)
- Actual configs are git-ignored (protect secrets)
- secrets.yaml is git-ignored (never commit)
4. **Cross-reference documentation:**
- Add links/references to:
- config/README.md (configuration reference)
- DEPLOYMENT.md (detailed deployment guide)
- dev_docs/ (for development setup)
5. **Add "Getting Help" section (if not exists):**
- Point to DEPLOYMENT.md troubleshooting
- Point to config/README.md for config questions
- Reference AI_RULES.md for project conventions
6. **Update any hardcoded inventory.env references:**
- Replace with config/ references
- Update any env-related docs/examples
7. **Maintain existing structure:**
- Don't delete or significantly reorder existing sections
- Just add/update config-related content and update cross-references
Keep README concise but informative. Detailed docs go in DEPLOYMENT.md and config/README.md.
</action>
<verify>
- `grep -q "config/" README.md` (references config structure)
- `grep -q "DEPLOYMENT\\.md\|config/README\\.md" README.md` (cross-references detailed docs)
- `grep -q "secrets.yaml\|git.*ignore" README.md || echo "pass"` (mentions secrets and gitignore, optional)
- File should be valid markdown: `grep "^#" README.md | head -3` (has headers)
</verify>
<done>
README.md updated with configuration management, quick setup steps, and documentation cross-references.
</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Git repository → Deployment | .gitignore must prevent secrets from being committed |
| Documentation → Users | Documentation must clearly explain security requirements |
| Environment → Container | Docker environment variables can expose secrets if logged |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-07-13 | Tampering | docker-compose volume mount | mitigate | Config volume mounted read-only `:ro`. Backend cannot modify config at runtime. Changes require host-level edits. |
| T-07-14 | Information Disclosure | DEPLOYMENT.md instructions | mitigate | Documentation warns to generate JWT_SECRET_KEY, don't use placeholder. Warns about secrets.yaml setup. |
| T-07-15 | Information Disclosure | .gitignore config rules | mitigate | Clear rules prevent accidental secret commits. !config/*.example exception ensures schema is tracked. |
| T-07-16 | Elevation of Privilege | Docker container permissions | mitigate | No RUN as root in Dockerfile. Container runs as unprivileged user (if specified in docker-compose). |
</threat_model>
<verification>
**Phase 7, Plan 4 Verification Checklist:**
1. **docker-compose.yml**
- [ ] inventory.env env_file removed or commented (per D-04)
- [ ] ./config:/app/config:ro volume mount added to backend service
- [ ] Syntax valid: `docker-compose config` succeeds
- [ ] Comments reference D-07, D-04 decisions
- [ ] Environment variables preserved (JWT_SECRET_KEY etc. with production warning)
2. **backend/Dockerfile**
- [ ] No references to inventory.env
- [ ] Comments reference config/ structure and D-07
- [ ] ENTRYPOINT or CMD properly set
- [ ] Syntax valid: `docker build --dry-run` or manual parse
3. **backend/entrypoint.sh**
- [ ] No sourcing of inventory.env
- [ ] References /app/config/ paths
- [ ] Checks if config/backend.yaml exists
- [ ] Documents environment variable override behavior
- [ ] Bash syntax valid: `bash -n backend/entrypoint.sh`
4. **.gitignore**
- [ ] Rules added: config/*.yaml, !config/*.yaml.example, config/secrets.yaml, !config/secrets.yaml.example
- [ ] Old inventory.env references removed or marked deprecated
- [ ] Git test: `git check-ignore config/backend.yaml` returns success (ignored)
- [ ] Git test: `git status config/*.example` shows untracked (not ignored)
5. **DEPLOYMENT.md**
- [ ] Configuration section added before or after Quick Start
- [ ] References config/ files (backend.yaml, frontend.yaml, network.yaml, docker.yaml, secrets.yaml)
- [ ] Docker deployment updated to use scripts/deploy.py
- [ ] Standalone deployment updated to use scripts/run_standalone.py
- [ ] Systemd service section with scripts/install_service.py
- [ ] Backup section with scripts/export_prod.py
- [ ] Troubleshooting section
- [ ] Migration section (from inventory.env to config/)
- [ ] Length 150+ lines
6. **README.md**
- [ ] Configuration section added
- [ ] Quick start updated with config setup steps
- [ ] Cross-references to config/README.md and DEPLOYMENT.md
- [ ] Mentions secrets.yaml and .gitignore
- [ ] No hardcoded inventory.env references
7. **Cross-document consistency**
- [ ] README.md, DEPLOYMENT.md, config/README.md use consistent terminology
- [ ] All three documents reference each other appropriately
- [ ] No conflicting instructions across documents
</verification>
<success_criteria>
- docker-compose.yml updated with config/ volume mount, inventory.env env_file removed (D-07)
- backend/Dockerfile and entrypoint.sh updated for new config paths
- .gitignore configured to track examples, ignore actual configs and secrets (D-08)
- DEPLOYMENT.md updated with YAML structure, Python scripts, systemd setup, troubleshooting (D-08)
- README.md updated with config setup and documentation cross-references (D-08)
- All documentation references config/ as single source of truth
- Docker deployment works with new config structure
- Clear migration path from inventory.env to new config/ structure documented
</success_criteria>
<output>
After completion, create `.planning/phases/07-config-consolidation/07-04-SUMMARY.md`
</output>