diff --git a/.planning/phases/07-config-consolidation/07-CONTEXT.md b/.planning/phases/07-config-consolidation/07-CONTEXT.md index 430ff73e..f95da1b9 100644 --- a/.planning/phases/07-config-consolidation/07-CONTEXT.md +++ b/.planning/phases/07-config-consolidation/07-CONTEXT.md @@ -30,73 +30,145 @@ Consolidate all application configuration files into a centralized `config/` fol ## Implementation Decisions -### Configuration Structure -- Create `config/` folder in project root as the central location for all configurations -- Use meaningful filenames: `backend.env`, `frontend.env`, `network.env`, `docker.env` (for docker-compose) -- Keep backward compatibility: Support both old (`inventory.env`) and new (`config/backend.env`) paths during transition -- Environment variable precedence: System env vars > Docker build args > config/backend.env > defaults in code +### D-01: Configuration File Format +- **Standardize on YAML format** for all config files (backend.yaml, frontend.yaml, network.yaml, docker.yaml) +- All config files in `config/` folder will be YAML format +- Backend code updated to use PyYAML parser +- Rationale: YAML provides better structure for complex configs, easier validation, clearer schema -### Backend Loading -- Update `backend/config_loader.py` to look for configs in `config/backend.env` first -- If not found, fall back to `inventory.env` (backward compatibility) -- If neither exists, use system environment variables +### D-02: Secrets Management (Separate File) +- Create dedicated `config/secrets.yaml` file for sensitive values (API keys, JWT secrets, database passwords) +- Add `config/secrets.yaml` to `.gitignore` with strict exclusion +- Commit `config/secrets.yaml.example` with placeholder values and clear format requirements +- Include strong documentation in config/README.md explaining each secret, where to obtain it, format requirements +- Rationale: Clear separation of concerns between configuration and secrets, guides developers on required values + +### D-03: Config File Examples +- Commit `.example` files for ALL config files: `backend.yaml.example`, `frontend.yaml.example`, `network.yaml.example`, `docker.yaml.example` +- Developers copy examples to non-example versions locally and fill in values +- Example files show structure, defaults, and all available options +- Rationale: Clear onboarding path, version control of config schema, consistency guarantees + +### D-04: Backward Compatibility - Immediate Deprecation +- **NO fallback to `inventory.env`** - immediate deprecation after Phase 7 completes +- All deployments must migrate to new `config/` structure during this phase +- Remove all code paths that read from root-level `inventory.env` +- Rationale: Clean break avoids ongoing dual-path support complexity + +### D-05: Deployment Scripts - Convert Bash to Python +- Convert all necessary bash deployment scripts to Python with identical functionality +- Before conversion: **Audit all scripts to identify redundant/mergeable ones** +- Critical scripts to convert: `deploy.sh`, `run_standalone.sh`, `install_service.sh`, `export_prod.sh` +- Evaluate `__push_ALL_to_remote.sh` for necessity/consolidation +- All Python scripts will parse YAML config files +- Rationale: Consistent tooling across infrastructure, easier YAML parsing, reduced bash complexity + +### D-06: Backend Config Loading +- Update `backend/config_loader.py` to parse YAML files +- Load order: System environment variables > `config/backend.yaml` > defaults in code +- Remove any fallback to `inventory.env` (Phase 7 end → fully deprecated) - Log which config source is being used for debugging -### Deployment Scripts -- `deploy.sh`: Source environment from `config/network.env` or `config/docker.env` instead of `inventory.env` -- `run_standalone.sh`: Source configuration from `config/backend.env` and `config/network.env` -- Docker Compose: Updated to reference `config/docker.env` -- Scripts should create symlinks or validate that config folder exists before running +### D-07: Docker & Docker Compose +- Docker Compose updated to reference `config/docker.yaml` +- Backend Dockerfile and frontend Dockerfile updated to source from new config structure +- Environment variable injection mechanism preserved (takes precedence over YAML files) -### Cleanup Actions -- Remove or archive `inventory.env` templates after migration -- Audit root directory scripts for redundancy -- Update .gitignore to ensure config/ folder structure is tracked (if needed) - -### Documentation Updates -- Update DEPLOYMENT.md with new config folder structure -- Update README.md with configuration setup instructions -- Add config/README.md explaining each config file's purpose +### D-08: Documentation & Git Structure +- Update DEPLOYMENT.md with new YAML config structure, required secrets, and format specifications +- Update README.md with configuration setup and onboarding instructions +- Add comprehensive `config/README.md` explaining all YAML files, required variables, examples, secrets setup +- Update .gitignore: ignore `config/*.yaml` (except examples), track `config/*.yaml.example` --- ## Specific Ideas -1. **Config Files to Create:** - - `config/backend.env` — Backend-specific variables (database, AI keys, auth settings) - - `config/frontend.env` — Frontend-specific variables (API endpoints, feature flags) - - `config/network.env` — Network/deployment variables (ports, SSL, server IPs) - - `config/docker.env` — Docker-specific overrides (for docker-compose.yml) - - `config/README.md` — Documentation of all configuration files +1. **YAML Config Files to Create:** + - `config/backend.yaml` — Backend-specific variables (database, AI keys, auth settings, logging) + - `config/backend.yaml.example` — Template showing all available options + - `config/frontend.yaml` — Frontend-specific variables (API endpoints, feature flags, service worker settings) + - `config/frontend.yaml.example` — Frontend config template + - `config/network.yaml` — Network/deployment variables (ports, SSL, server IPs, CORS settings) + - `config/network.yaml.example` — Network config template + - `config/docker.yaml` — Docker-specific overrides (for docker-compose.yml) + - `config/docker.yaml.example` — Docker config template + - `config/secrets.yaml` — Sensitive values (git-ignored) + - `config/secrets.yaml.example` — Secrets template with placeholders + - `config/README.md` — Comprehensive documentation of all YAML files, structure, required values -2. **Scripts to Update:** - - `deploy.sh` — Update to source from config/ folder - - `run_standalone.sh` — Update network config loading - - `export_prod.sh` — Ensure it uses new config structure - - `install_service.sh` — Update systemd service to reference new config paths - - `__push_ALL_to_remote.sh` — Review for config-related updates +2. **Python Scripts to Create (replacing bash):** + - `scripts/deploy.py` — Docker deployment with YAML config parsing (replaces deploy.sh) + - `scripts/run_standalone.py` — Standalone mode launcher (replaces run_standalone.sh) + - `scripts/export_prod.py` — Production export/backup functionality + - `scripts/install_service.py` — Systemd service installation (replaces install_service.sh) + - Audit `__push_ALL_to_remote.sh` - determine if needed or consolidate into another script + - All scripts will use PyYAML for config file parsing 3. **Backend Changes:** - - `backend/config_loader.py` — Update load order: config/backend.env > inventory.env > env vars - - `backend/config_manager.py` — Update to read/write from config/backend.env - - `backend/entrypoint.sh` — Source from config/ instead of root level + - `backend/config_loader.py` — Update to parse YAML files (backend.yaml + secrets.yaml) + - Load order: System env vars > config/backend.yaml > config/secrets.yaml > defaults in code + - Remove all code paths for reading inventory.env + - Implement environment variable override mechanism (system env vars take precedence) + - `backend/config_manager.py` — Update to read/write YAML (if config updates are needed at runtime) + - `backend/entrypoint.sh` — Reference new config paths in container 4. **Testing Requirements:** - - Docker deployment with new config structure - - Standalone deployment with new config structure - - Environment variable override behavior still works - - Backward compatibility: old inventory.env still loads if config/ doesn't exist + - Docker deployment with YAML config structure + - Standalone Python launcher with YAML config parsing + - Environment variable override behavior with YAML configs + - Secrets file permissions and git-ignore verification + - All deployment paths tested end-to-end with new Python scripts + - Confirm old inventory.env paths are NOT accessible (no fallback) + +--- + +## Claude's Discretion + +- Structure and organization of Python scripts in `scripts/` folder +- YAML validation schema and enforcement approach (basic vs. strict validation) +- Logging verbosity and format in Python deployment scripts + +--- + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Configuration & Deployment +- `DEPLOYMENT.md` — Current deployment procedures (to be updated with YAML structure) +- `README.md` — Project setup instructions (to be updated with config onboarding) +- `PROJECT_ARCHITECTURE.md` — Technical stack and component overview + +### Backend Configuration Loading +- `backend/config_loader.py` — Current config loading implementation (will be refactored for YAML) +- `backend/config_manager.py` — Current config management (will be updated) + +### Deployment Scripts (to be rewritten in Python) +- `deploy.sh` — Docker deployment script +- `run_standalone.sh` — Standalone mode launcher +- `export_prod.sh` — Production export +- `install_service.sh` — Systemd service setup +- `__push_ALL_to_remote.sh` — Remote push utility (to be audited for necessity) + +### Infrastructure Files +- `docker-compose.yml` — Docker composition (to be updated to reference config/docker.yaml) +- `backend/Dockerfile` — Backend container image (to be updated for YAML config paths) +- `frontend/Dockerfile` — Frontend container image (if applicable) +- `inventory.env` — Current config location (will be deprecated after Phase 7) --- ## Deferred Ideas -- Dynamic config hot-reload without restart (future optimization) -- Config validation framework (future enhancement) -- Encrypted sensitive values in config files (future security enhancement) -- Web UI for configuration management (future feature) +- Dynamic config hot-reload without restart (future optimization — Phase N) +- Config validation framework with JSON Schema (future enhancement — Phase N) +- Encrypted sensitive values in config files via KMS or Vault (future security enhancement — Phase N+1) +- Web UI for configuration management (future feature — Phase N+2) +- Config versioning and rollback mechanism (future ops enhancement) --- *Phase: 7-config-consolidation* -*Context gathered: 2026-04-23 from user requirements* +*Context gathered: 2026-04-23 via structured discussion* +*Status: Ready for detailed planning with locked decisions* diff --git a/.planning/phases/07-config-consolidation/07-DISCUSSION-LOG.md b/.planning/phases/07-config-consolidation/07-DISCUSSION-LOG.md new file mode 100644 index 00000000..9ca16f01 --- /dev/null +++ b/.planning/phases/07-config-consolidation/07-DISCUSSION-LOG.md @@ -0,0 +1,158 @@ +# Phase 7: Config Consolidation - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-04-23 +**Phase:** 7-config-consolidation +**Areas discussed:** Secrets Management, Config File Examples, Backward Compatibility Timeline, Config File Format & Scripts + +--- + +## 1. Secrets & Sensitive Values + +**Question:** How should sensitive values (API keys, JWT secrets, database passwords) be managed in the config structure? + +| Option | Description | Selected | +|--------|-------------|----------| +| Store in config/backend.env (git-ignored) | Developers create their own local config/backend.env with secrets. File is in .gitignore so secrets never reach git. Example files committed separately for reference. | | +| Environment variables only | No sensitive values in config files. Deploy processes inject secrets via OS environment variables (Docker secrets, systemd, k8s secrets). Config files contain only non-sensitive settings. | | +| Separate secrets file | Create config/secrets.env (separate from backend.env) with stricter .gitignore rules. Non-sensitive backend config still in backend.env. Clearer separation of concerns. | ✓ | +| External secrets manager reference | Config files contain references/URLs to external secret storage (HashiCorp Vault, AWS Secrets Manager, etc.). Application loads secrets at runtime from external source. | | + +**User's choice:** Separate secrets file +**Notes:** Clear separation of concerns. Dedicated file for sensitive values with strict handling. + +--- + +## 1b. Secrets Template Approach + +**Question:** Should config/secrets.env.example be committed to git to guide developers on required secrets? + +| Option | Description | Selected | +|--------|-------------|----------| +| Yes, commit example with placeholder values | config/secrets.env.example in git (not git-ignored) shows all required secrets and example formats. Developers copy it to config/secrets.env and fill in real values locally. | | +| No, document in README only | No example file in git. Required secrets documented in config/README.md and DEPLOYMENT.md. Reduces git noise, relies on good documentation. | | +| Both — example + strong documentation | Commit config/secrets.env.example AND provide detailed docs explaining each secret, where to get it, format requirements. Best for onboarding but more maintenance. | ✓ | + +**User's choice:** Both — example + strong documentation +**Notes:** Provides clear onboarding path for developers while maintaining comprehensive documentation. + +--- + +## 2. Config File Format & Examples + +**Question:** How should non-sensitive config file examples be managed? + +| Option | Description | Selected | +|--------|-------------|----------| +| Commit all .example files | All config files have .example variants in git. Developers copy them locally. Easy onboarding, version control of all config schemas. | ✓ | +| Only document in config/README.md | No .example files. config/README.md lists all variables, defaults, and descriptions. Cleaner git repo, single source of truth is docs. | | +| Template files (versioned, not copyrighted) | Commit minimal template files with just variable names (no values). Like .example but more concise. Developers fill in values. Balance between clarity and repo cleanliness. | | +| Docker compose example in separate file | Only docker.env.example committed (since it's tied to docker-compose.yml). Backend/frontend examples in docs only. Minimal git footprint for typical deployments. | | + +**User's choice:** Commit all .example files +**Notes:** Developer-friendly approach. Clear templates to copy and fill in. + +--- + +## 3. Backward Compatibility Timeline + +**Question:** How long should the system support loading from old inventory.env as fallback? + +| Option | Description | Selected | +|--------|-------------|----------| +| This phase only (immediate deprecation) | After Phase 7 completes, inventory.env support is removed entirely. Requires all deployments to migrate immediately. Cleanest going forward, but higher migration burden. | ✓ | +| 2-3 releases (gradual sunset) | Support old paths for next 2-3 releases. Deprecated in logging. Then removed. Gives production deployments time to migrate gracefully. | | +| Keep indefinitely for backward compatibility | Never remove old paths. Scripts support both config/ and inventory.env forever. Highest compatibility, but code complexity persists. | | + +**User's choice:** This phase only (immediate deprecation) +**Notes:** Clean break. All deployments must migrate during Phase 7. No ongoing dual-path complexity. + +--- + +## 4. Config File Format & Deployment Scripts + +**Question:** What file format should the consolidated config files use? + +| Option | Description | Selected | +|--------|-------------|----------| +| Keep .env format (minimal change) | Stick with KEY=VALUE .env files already used by python-dotenv. Least disruptive. Backend and deployment scripts already know how to read this. | | +| Standardize on .yaml (structured but verbose setup) | Migrate to YAML format. More structured, easier to validate, better for complex configs. Requires adding pyyaml dependency, updating all loaders. | ✓ | +| Use .json (most rigid, easiest parsing) | JSON format for all configs. Very structured, standard parsing libraries, but less friendly for manual editing and comments. | | +| Multi-format support (backend handles both) | Support both .env AND .yaml/.json. Loader auto-detects format. Flexibility, but adds parsing complexity and documentation burden. | | + +**User's choice:** Standardize on .yaml +**Notes:** More structured format. Better for complex configurations and validation. + +--- + +## 4b. YAML Configuration in Deployment Scripts + +**Question:** For YAML-based config, how should deployment scripts (bash) access configuration? + +| Option | Description | Selected | +|--------|-------------|----------| +| Scripts read .env files, backend reads YAML — dual formats | Keep scripts using .env (familiar bash tooling). Backend uses YAML. Translation/sync between the two. More maintenance, but each tool uses its native format. | | +| Export YAML to env vars, scripts use vars | Config loading converts YAML to environment variables before scripts run. Scripts source env vars. Cleaner separation, requires loader to run first. | | +| Convert to .yaml everywhere, write yaml parser for scripts | Everything uses YAML. Write or use lightweight YAML parser in bash (yq). Consistent format end-to-end, adds bash dependency. | | +| Keep .env for network/docker, YAML for backend only | config/network.env and config/docker.env stay as .env (for scripts). Only config/backend.yaml uses YAML. Hybrid approach based on where each is used. | | + +**User's choice (custom):** Convert all config to YAML files, and convert all bash deployment scripts to Python scripts +**Notes:** Comprehensive modernization. Replace bash scripts with Python equivalents that natively parse YAML. More significant scope, but cleaner end-to-end architecture. + +--- + +## 4c. Script Conversion Scope + +**Question:** Is script conversion (bash → Python) part of Phase 7 scope, or a follow-up phase? + +| Option | Description | Selected | +|--------|-------------|----------| +| Include in Phase 7 | Rewrite all deployment/utility scripts in Python as part of this phase. Phase 7 scope expands significantly but delivers complete Python-based stack. | ✓ | +| Phase 7: YAML config only, follow-up phase for scripts | Phase 7 focuses on YAML config consolidation. Bash scripts adapted to read YAML (via yq or similar). Script conversion deferred to a future phase. | | +| Hybrid approach this phase | Phase 7: Convert the critical deployment scripts (deploy.sh, run_standalone.sh) to Python. Defer non-critical scripts (export_prod.sh, __push_ALL_to_remote.sh) to future phase. | | + +**User's choice:** Include in Phase 7 +**Notes:** Comprehensive Phase 7 scope. All necessary Python scripts completed in this phase. + +--- + +## 4d. Script Priority & Consolidation + +**Question:** Which scripts are critical for Phase 7 conversion? Which can be deferred? + +**User's choice (custom):** Audit all bash scripts to identify redundant/mergeable ones. Convert all needed scripts to Python. +**Notes:** Consolidation-focused approach. Before converting, identify opportunities to merge redundant functionality and eliminate unnecessary scripts. Then convert only what's essential. + +--- + +## Summary of Locked Decisions + +- **D-01:** YAML format for all config files (backend.yaml, frontend.yaml, network.yaml, docker.yaml) +- **D-02:** Separate secrets.yaml file (git-ignored) + secrets.yaml.example (committed) +- **D-03:** Commit all .example files for config schema reference +- **D-04:** Immediate deprecation of inventory.env (no fallback after Phase 7) +- **D-05:** Convert all necessary bash deployment scripts to Python +- **D-06:** Audit scripts first to consolidate redundancy before conversion + +--- + +## Claude's Discretion + +Areas where the user deferred to Claude's judgment: +- Structure and organization of Python scripts in `scripts/` folder +- YAML validation schema and enforcement approach +- Logging verbosity and format in Python deployment scripts + +--- + +## Deferred Ideas + +(None mentioned during discussion) + +--- + +*Discussion conducted: 2026-04-23* +*Format: Structured Q&A with alternatives considered* +*Outcome: All gray areas resolved; ready for detailed planning*