bede/tfm_ainventory
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(phase-7): capture consolidated context from structured discussion

Browse Source 
Discussion locked in 4 key decisions:

1. YAML Format: All configs standardized on YAML (backend.yaml, frontend.yaml,
   network.yaml, docker.yaml) for better structure and validation

2. Secrets Management: Separate config/secrets.yaml file (git-ignored) with
   config/secrets.yaml.example template + strong documentation

3. Examples: Commit all .yaml.example files for developer onboarding

4. Scripts: Audit bash scripts for redundancy, then convert all needed ones
   to Python for native YAML parsing and cleaner architecture

Immediate deprecation of inventory.env (no fallback after Phase 7).

Discussion log preserved in 07-DISCUSSION-LOG.md

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
Daniel Bedeleanu
2026-04-23 12:20:53 +03:00
parent 2b7a2b3a89
commit dacf7d2921
2 changed files with 277 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

166
.planning/phases/07-config-consolidation/07-CONTEXT.md
View File

@@ -30,73 +30,145 @@ Consolidate all application configuration files into a centralized `config/` fol
## Implementation Decisions
### Configuration Structure
- Create `config/` folder in project root as the central location for all configurations
- Use meaningful filenames: `backend.env`, `frontend.env`, `network.env`, `docker.env` (for docker-compose)
- Keep backward compatibility: Support both old (`inventory.env`) and new (`config/backend.env`) paths during transition
- Environment variable precedence: System env vars > Docker build args > config/backend.env > defaults in code
### D-01: Configuration File Format
- **Standardize on YAML format** for all config files (backend.yaml, frontend.yaml, network.yaml, docker.yaml)
- All config files in `config/` folder will be YAML format
- Backend code updated to use PyYAML parser
- Rationale: YAML provides better structure for complex configs, easier validation, clearer schema
### Backend Loading
- Update `backend/config_loader.py` to look for configs in `config/backend.env` first
- If not found, fall back to `inventory.env` (backward compatibility)
- If neither exists, use system environment variables
### D-02: Secrets Management (Separate File)
- Create dedicated `config/secrets.yaml` file for sensitive values (API keys, JWT secrets, database passwords)
- Add `config/secrets.yaml` to `.gitignore` with strict exclusion
- Commit `config/secrets.yaml.example` with placeholder values and clear format requirements
- Include strong documentation in config/README.md explaining each secret, where to obtain it, format requirements
- Rationale: Clear separation of concerns between configuration and secrets, guides developers on required values
### D-03: Config File Examples
- Commit `.example` files for ALL config files: `backend.yaml.example`, `frontend.yaml.example`, `network.yaml.example`, `docker.yaml.example`
- Developers copy examples to non-example versions locally and fill in values
- Example files show structure, defaults, and all available options
- Rationale: Clear onboarding path, version control of config schema, consistency guarantees
### D-04: Backward Compatibility - Immediate Deprecation
- **NO fallback to `inventory.env`** - immediate deprecation after Phase 7 completes
- All deployments must migrate to new `config/` structure during this phase
- Remove all code paths that read from root-level `inventory.env`
- Rationale: Clean break avoids ongoing dual-path support complexity
### D-05: Deployment Scripts - Convert Bash to Python
- Convert all necessary bash deployment scripts to Python with identical functionality
- Before conversion: **Audit all scripts to identify redundant/mergeable ones**
- Critical scripts to convert: `deploy.sh`, `run_standalone.sh`, `install_service.sh`, `export_prod.sh`
- Evaluate `__push_ALL_to_remote.sh` for necessity/consolidation
- All Python scripts will parse YAML config files
- Rationale: Consistent tooling across infrastructure, easier YAML parsing, reduced bash complexity
### D-06: Backend Config Loading
- Update `backend/config_loader.py` to parse YAML files
- Load order: System environment variables > `config/backend.yaml` > defaults in code
- Remove any fallback to `inventory.env` (Phase 7 end → fully deprecated)
- Log which config source is being used for debugging
### Deployment Scripts
- `deploy.sh`: Source environment from `config/network.env` or `config/docker.env` instead of `inventory.env`
- `run_standalone.sh`: Source configuration from `config/backend.env` and `config/network.env`
- Docker Compose: Updated to reference `config/docker.env`
- Scripts should create symlinks or validate that config folder exists before running
### D-07: Docker & Docker Compose
- Docker Compose updated to reference `config/docker.yaml`
- Backend Dockerfile and frontend Dockerfile updated to source from new config structure
- Environment variable injection mechanism preserved (takes precedence over YAML files)
### Cleanup Actions
- Remove or archive `inventory.env` templates after migration
- Audit root directory scripts for redundancy
- Update .gitignore to ensure config/ folder structure is tracked (if needed)
### Documentation Updates
- Update DEPLOYMENT.md with new config folder structure
- Update README.md with configuration setup instructions
- Add config/README.md explaining each config file's purpose
### D-08: Documentation & Git Structure
- Update DEPLOYMENT.md with new YAML config structure, required secrets, and format specifications
- Update README.md with configuration setup and onboarding instructions
- Add comprehensive `config/README.md` explaining all YAML files, required variables, examples, secrets setup
- Update .gitignore: ignore `config/*.yaml` (except examples), track `config/*.yaml.example`
---
## Specific Ideas
1. **Config Files to Create:**
- `config/backend.env` — Backend-specific variables (database, AI keys, auth settings)
- `config/frontend.env` — Frontend-specific variables (API endpoints, feature flags)
- `config/network.env` — Network/deployment variables (ports, SSL, server IPs)
- `config/docker.env` — Docker-specific overrides (for docker-compose.yml)
- `config/README.md` — Documentation of all configuration files
1. **YAML Config Files to Create:**
- `config/backend.yaml` — Backend-specific variables (database, AI keys, auth settings, logging)
- `config/backend.yaml.example` — Template showing all available options
- `config/frontend.yaml` — Frontend-specific variables (API endpoints, feature flags, service worker settings)
- `config/frontend.yaml.example` — Frontend config template
- `config/network.yaml` — Network/deployment variables (ports, SSL, server IPs, CORS settings)
- `config/network.yaml.example` — Network config template
- `config/docker.yaml` — Docker-specific overrides (for docker-compose.yml)
- `config/docker.yaml.example` — Docker config template
- `config/secrets.yaml` — Sensitive values (git-ignored)
- `config/secrets.yaml.example` — Secrets template with placeholders
- `config/README.md` — Comprehensive documentation of all YAML files, structure, required values
2. **Scripts to Update:**
- `deploy.sh` — Update to source from config/ folder
- `run_standalone.sh` — Update network config loading
- `export_prod.sh` — Ensure it uses new config structure
- `install_service.sh` — Update systemd service to reference new config paths
- `__push_ALL_to_remote.sh` — Review for config-related updates
2. **Python Scripts to Create (replacing bash):**
- `scripts/deploy.py` — Docker deployment with YAML config parsing (replaces deploy.sh)
- `scripts/run_standalone.py` — Standalone mode launcher (replaces run_standalone.sh)
- `scripts/export_prod.py` — Production export/backup functionality
- `scripts/install_service.py` — Systemd service installation (replaces install_service.sh)
- Audit `__push_ALL_to_remote.sh` - determine if needed or consolidate into another script
- All scripts will use PyYAML for config file parsing
3. **Backend Changes:**
- `backend/config_loader.py` — Update load order: config/backend.env > inventory.env > env vars
- `backend/config_manager.py` — Update to read/write from config/backend.env
- `backend/entrypoint.sh` — Source from config/ instead of root level
- `backend/config_loader.py` — Update to parse YAML files (backend.yaml + secrets.yaml)
- Load order: System env vars > config/backend.yaml > config/secrets.yaml > defaults in code
- Remove all code paths for reading inventory.env
- Implement environment variable override mechanism (system env vars take precedence)
- `backend/config_manager.py` — Update to read/write YAML (if config updates are needed at runtime)
- `backend/entrypoint.sh` — Reference new config paths in container
4. **Testing Requirements:**
- Docker deployment with new config structure
- Standalone deployment with new config structure
- Environment variable override behavior still works
- Backward compatibility: old inventory.env still loads if config/ doesn't exist
- Docker deployment with YAML config structure
- Standalone Python launcher with YAML config parsing
- Environment variable override behavior with YAML configs
- Secrets file permissions and git-ignore verification
- All deployment paths tested end-to-end with new Python scripts
- Confirm old inventory.env paths are NOT accessible (no fallback)
---
## Claude's Discretion
- Structure and organization of Python scripts in `scripts/` folder
- YAML validation schema and enforcement approach (basic vs. strict validation)
- Logging verbosity and format in Python deployment scripts
---
## Canonical References
**Downstream agents MUST read these before planning or implementing.**
### Configuration & Deployment
- `DEPLOYMENT.md` — Current deployment procedures (to be updated with YAML structure)
- `README.md` — Project setup instructions (to be updated with config onboarding)
- `PROJECT_ARCHITECTURE.md` — Technical stack and component overview
### Backend Configuration Loading
- `backend/config_loader.py` — Current config loading implementation (will be refactored for YAML)
- `backend/config_manager.py` — Current config management (will be updated)
### Deployment Scripts (to be rewritten in Python)
- `deploy.sh` — Docker deployment script
- `run_standalone.sh` — Standalone mode launcher
- `export_prod.sh` — Production export
- `install_service.sh` — Systemd service setup
- `__push_ALL_to_remote.sh` — Remote push utility (to be audited for necessity)
### Infrastructure Files
- `docker-compose.yml` — Docker composition (to be updated to reference config/docker.yaml)
- `backend/Dockerfile` — Backend container image (to be updated for YAML config paths)
- `frontend/Dockerfile` — Frontend container image (if applicable)
- `inventory.env` — Current config location (will be deprecated after Phase 7)
---
## Deferred Ideas
- Dynamic config hot-reload without restart (future optimization)
- Config validation framework (future enhancement)
- Encrypted sensitive values in config files (future security enhancement)
- Web UI for configuration management (future feature)
- Dynamic config hot-reload without restart (future optimization — Phase N)
- Config validation framework with JSON Schema (future enhancement — Phase N)
- Encrypted sensitive values in config files via KMS or Vault (future security enhancement — Phase N+1)
- Web UI for configuration management (future feature — Phase N+2)
- Config versioning and rollback mechanism (future ops enhancement)
---
*Phase: 7-config-consolidation*
*Context gathered: 2026-04-23 from user requirements*
*Context gathered: 2026-04-23 via structured discussion*
*Status: Ready for detailed planning with locked decisions*

158
.planning/phases/07-config-consolidation/07-DISCUSSION-LOG.md Normal file
View File

@@ -0,0 +1,158 @@
# Phase 7: Config Consolidation - Discussion Log
> **Audit trail only.** Do not use as input to planning, research, or execution agents.
> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
**Date:** 2026-04-23
**Phase:** 7-config-consolidation
**Areas discussed:** Secrets Management, Config File Examples, Backward Compatibility Timeline, Config File Format & Scripts
---
## 1. Secrets & Sensitive Values
**Question:** How should sensitive values (API keys, JWT secrets, database passwords) be managed in the config structure?
| Option | Description | Selected |
|--------|-------------|----------|
| Store in config/backend.env (git-ignored) | Developers create their own local config/backend.env with secrets. File is in .gitignore so secrets never reach git. Example files committed separately for reference. | |
| Environment variables only | No sensitive values in config files. Deploy processes inject secrets via OS environment variables (Docker secrets, systemd, k8s secrets). Config files contain only non-sensitive settings. | |
| Separate secrets file | Create config/secrets.env (separate from backend.env) with stricter .gitignore rules. Non-sensitive backend config still in backend.env. Clearer separation of concerns. | ✓ |
| External secrets manager reference | Config files contain references/URLs to external secret storage (HashiCorp Vault, AWS Secrets Manager, etc.). Application loads secrets at runtime from external source. | |
**User's choice:** Separate secrets file
**Notes:** Clear separation of concerns. Dedicated file for sensitive values with strict handling.
---
## 1b. Secrets Template Approach
**Question:** Should config/secrets.env.example be committed to git to guide developers on required secrets?
| Option | Description | Selected |
|--------|-------------|----------|
| Yes, commit example with placeholder values | config/secrets.env.example in git (not git-ignored) shows all required secrets and example formats. Developers copy it to config/secrets.env and fill in real values locally. | |
| No, document in README only | No example file in git. Required secrets documented in config/README.md and DEPLOYMENT.md. Reduces git noise, relies on good documentation. | |
| Both — example + strong documentation | Commit config/secrets.env.example AND provide detailed docs explaining each secret, where to get it, format requirements. Best for onboarding but more maintenance. | ✓ |
**User's choice:** Both — example + strong documentation
**Notes:** Provides clear onboarding path for developers while maintaining comprehensive documentation.
---
## 2. Config File Format & Examples
**Question:** How should non-sensitive config file examples be managed?
| Option | Description | Selected |
|--------|-------------|----------|
| Commit all .example files | All config files have .example variants in git. Developers copy them locally. Easy onboarding, version control of all config schemas. | ✓ |
| Only document in config/README.md | No .example files. config/README.md lists all variables, defaults, and descriptions. Cleaner git repo, single source of truth is docs. | |
| Template files (versioned, not copyrighted) | Commit minimal template files with just variable names (no values). Like .example but more concise. Developers fill in values. Balance between clarity and repo cleanliness. | |
| Docker compose example in separate file | Only docker.env.example committed (since it's tied to docker-compose.yml). Backend/frontend examples in docs only. Minimal git footprint for typical deployments. | |
**User's choice:** Commit all .example files
**Notes:** Developer-friendly approach. Clear templates to copy and fill in.
---
## 3. Backward Compatibility Timeline
**Question:** How long should the system support loading from old inventory.env as fallback?
| Option | Description | Selected |
|--------|-------------|----------|
| This phase only (immediate deprecation) | After Phase 7 completes, inventory.env support is removed entirely. Requires all deployments to migrate immediately. Cleanest going forward, but higher migration burden. | ✓ |
| 2-3 releases (gradual sunset) | Support old paths for next 2-3 releases. Deprecated in logging. Then removed. Gives production deployments time to migrate gracefully. | |
| Keep indefinitely for backward compatibility | Never remove old paths. Scripts support both config/ and inventory.env forever. Highest compatibility, but code complexity persists. | |
**User's choice:** This phase only (immediate deprecation)
**Notes:** Clean break. All deployments must migrate during Phase 7. No ongoing dual-path complexity.
---
## 4. Config File Format & Deployment Scripts
**Question:** What file format should the consolidated config files use?
| Option | Description | Selected |
|--------|-------------|----------|
| Keep .env format (minimal change) | Stick with KEY=VALUE .env files already used by python-dotenv. Least disruptive. Backend and deployment scripts already know how to read this. | |
| Standardize on .yaml (structured but verbose setup) | Migrate to YAML format. More structured, easier to validate, better for complex configs. Requires adding pyyaml dependency, updating all loaders. | ✓ |
| Use .json (most rigid, easiest parsing) | JSON format for all configs. Very structured, standard parsing libraries, but less friendly for manual editing and comments. | |
| Multi-format support (backend handles both) | Support both .env AND .yaml/.json. Loader auto-detects format. Flexibility, but adds parsing complexity and documentation burden. | |
**User's choice:** Standardize on .yaml
**Notes:** More structured format. Better for complex configurations and validation.
---
## 4b. YAML Configuration in Deployment Scripts
**Question:** For YAML-based config, how should deployment scripts (bash) access configuration?
| Option | Description | Selected |
|--------|-------------|----------|
| Scripts read .env files, backend reads YAML — dual formats | Keep scripts using .env (familiar bash tooling). Backend uses YAML. Translation/sync between the two. More maintenance, but each tool uses its native format. | |
| Export YAML to env vars, scripts use vars | Config loading converts YAML to environment variables before scripts run. Scripts source env vars. Cleaner separation, requires loader to run first. | |
| Convert to .yaml everywhere, write yaml parser for scripts | Everything uses YAML. Write or use lightweight YAML parser in bash (yq). Consistent format end-to-end, adds bash dependency. | |
| Keep .env for network/docker, YAML for backend only | config/network.env and config/docker.env stay as .env (for scripts). Only config/backend.yaml uses YAML. Hybrid approach based on where each is used. | |
**User's choice (custom):** Convert all config to YAML files, and convert all bash deployment scripts to Python scripts
**Notes:** Comprehensive modernization. Replace bash scripts with Python equivalents that natively parse YAML. More significant scope, but cleaner end-to-end architecture.
---
## 4c. Script Conversion Scope
**Question:** Is script conversion (bash → Python) part of Phase 7 scope, or a follow-up phase?
| Option | Description | Selected |
|--------|-------------|----------|
| Include in Phase 7 | Rewrite all deployment/utility scripts in Python as part of this phase. Phase 7 scope expands significantly but delivers complete Python-based stack. | ✓ |
| Phase 7: YAML config only, follow-up phase for scripts | Phase 7 focuses on YAML config consolidation. Bash scripts adapted to read YAML (via yq or similar). Script conversion deferred to a future phase. | |
| Hybrid approach this phase | Phase 7: Convert the critical deployment scripts (deploy.sh, run_standalone.sh) to Python. Defer non-critical scripts (export_prod.sh, __push_ALL_to_remote.sh) to future phase. | |
**User's choice:** Include in Phase 7
**Notes:** Comprehensive Phase 7 scope. All necessary Python scripts completed in this phase.
---
## 4d. Script Priority & Consolidation
**Question:** Which scripts are critical for Phase 7 conversion? Which can be deferred?
**User's choice (custom):** Audit all bash scripts to identify redundant/mergeable ones. Convert all needed scripts to Python.
**Notes:** Consolidation-focused approach. Before converting, identify opportunities to merge redundant functionality and eliminate unnecessary scripts. Then convert only what's essential.
---
## Summary of Locked Decisions
- **D-01:** YAML format for all config files (backend.yaml, frontend.yaml, network.yaml, docker.yaml)
- **D-02:** Separate secrets.yaml file (git-ignored) + secrets.yaml.example (committed)
- **D-03:** Commit all .example files for config schema reference
- **D-04:** Immediate deprecation of inventory.env (no fallback after Phase 7)
- **D-05:** Convert all necessary bash deployment scripts to Python
- **D-06:** Audit scripts first to consolidate redundancy before conversion
---
## Claude's Discretion
Areas where the user deferred to Claude's judgment:
- Structure and organization of Python scripts in `scripts/` folder
- YAML validation schema and enforcement approach
- Logging verbosity and format in Python deployment scripts
---
## Deferred Ideas
(None mentioned during discussion)
---
*Discussion conducted: 2026-04-23*
*Format: Structured Q&A with alternatives considered*
*Outcome: All gray areas resolved; ready for detailed planning*