From 2b7a2b3a8910eb212faedf58a9f31fd44b7d9dc6 Mon Sep 17 00:00:00 2001 From: Daniel Bedeleanu Date: Thu, 23 Apr 2026 12:07:39 +0300 Subject: [PATCH] docs(phase-7): add configuration consolidation planning - Phase 7: Config Consolidation - CONTEXT.md: Defines scope, decisions, and requirements - PLAN.md: 18-task implementation plan with detailed acceptance criteria - Centralize all configs into config/ folder - Update deploy scripts, backend loading, and documentation - Maintain backward compatibility with inventory.env Co-Authored-By: Claude Haiku 4.5 --- .../07-config-consolidation/07-CONTEXT.md | 102 ++++ .../phases/07-config-consolidation/07-PLAN.md | 502 ++++++++++++++++++ 2 files changed, 604 insertions(+) create mode 100644 .planning/phases/07-config-consolidation/07-CONTEXT.md create mode 100644 .planning/phases/07-config-consolidation/07-PLAN.md diff --git a/.planning/phases/07-config-consolidation/07-CONTEXT.md b/.planning/phases/07-config-consolidation/07-CONTEXT.md new file mode 100644 index 00000000..430ff73e --- /dev/null +++ b/.planning/phases/07-config-consolidation/07-CONTEXT.md @@ -0,0 +1,102 @@ +# Phase 7: Config Consolidation - Context + +**Gathered:** 2026-04-23 +**Status:** Ready for planning +**Source:** User Requirements + +--- + +## Phase Boundary + +Consolidate all application configuration files into a centralized `config/` folder in the project root. This includes backend configurations, frontend configurations, deployment settings, and network configurations. Update all deployment scripts, application startup procedures, and backend/frontend code to load configurations from this centralized location. + +**Scope:** +- Create and establish `config/` folder as the single source of truth for all application configuration +- Migrate existing configuration files (inventory.env and variants) to config folder with meaningful names +- Update all scripts (deploy.sh, run_standalone.sh, etc.) to reference the new config location +- Refactor backend config_loader.py and config_manager.py to read from config folder +- Update frontend environment loading if applicable +- Verify and clean up root directory scripts that are no longer needed +- Ensure Docker deployment, standalone deployment, and development all work correctly with new structure + +**Deliverables:** +- `config/` folder with structured configuration files +- Updated backend configuration loading mechanism +- Updated deployment scripts +- Updated startup procedures (Docker and standalone) +- Documentation of configuration structure in README/DEPLOYMENT.md + +--- + +## Implementation Decisions + +### Configuration Structure +- Create `config/` folder in project root as the central location for all configurations +- Use meaningful filenames: `backend.env`, `frontend.env`, `network.env`, `docker.env` (for docker-compose) +- Keep backward compatibility: Support both old (`inventory.env`) and new (`config/backend.env`) paths during transition +- Environment variable precedence: System env vars > Docker build args > config/backend.env > defaults in code + +### Backend Loading +- Update `backend/config_loader.py` to look for configs in `config/backend.env` first +- If not found, fall back to `inventory.env` (backward compatibility) +- If neither exists, use system environment variables +- Log which config source is being used for debugging + +### Deployment Scripts +- `deploy.sh`: Source environment from `config/network.env` or `config/docker.env` instead of `inventory.env` +- `run_standalone.sh`: Source configuration from `config/backend.env` and `config/network.env` +- Docker Compose: Updated to reference `config/docker.env` +- Scripts should create symlinks or validate that config folder exists before running + +### Cleanup Actions +- Remove or archive `inventory.env` templates after migration +- Audit root directory scripts for redundancy +- Update .gitignore to ensure config/ folder structure is tracked (if needed) + +### Documentation Updates +- Update DEPLOYMENT.md with new config folder structure +- Update README.md with configuration setup instructions +- Add config/README.md explaining each config file's purpose + +--- + +## Specific Ideas + +1. **Config Files to Create:** + - `config/backend.env` — Backend-specific variables (database, AI keys, auth settings) + - `config/frontend.env` — Frontend-specific variables (API endpoints, feature flags) + - `config/network.env` — Network/deployment variables (ports, SSL, server IPs) + - `config/docker.env` — Docker-specific overrides (for docker-compose.yml) + - `config/README.md` — Documentation of all configuration files + +2. **Scripts to Update:** + - `deploy.sh` — Update to source from config/ folder + - `run_standalone.sh` — Update network config loading + - `export_prod.sh` — Ensure it uses new config structure + - `install_service.sh` — Update systemd service to reference new config paths + - `__push_ALL_to_remote.sh` — Review for config-related updates + +3. **Backend Changes:** + - `backend/config_loader.py` — Update load order: config/backend.env > inventory.env > env vars + - `backend/config_manager.py` — Update to read/write from config/backend.env + - `backend/entrypoint.sh` — Source from config/ instead of root level + +4. **Testing Requirements:** + - Docker deployment with new config structure + - Standalone deployment with new config structure + - Environment variable override behavior still works + - Backward compatibility: old inventory.env still loads if config/ doesn't exist + +--- + +## Deferred Ideas + +- Dynamic config hot-reload without restart (future optimization) +- Config validation framework (future enhancement) +- Encrypted sensitive values in config files (future security enhancement) +- Web UI for configuration management (future feature) + +--- + +*Phase: 7-config-consolidation* +*Context gathered: 2026-04-23 from user requirements* diff --git a/.planning/phases/07-config-consolidation/07-PLAN.md b/.planning/phases/07-config-consolidation/07-PLAN.md new file mode 100644 index 00000000..56d3f9cd --- /dev/null +++ b/.planning/phases/07-config-consolidation/07-PLAN.md @@ -0,0 +1,502 @@ +--- +wave: 1 +depends_on: [] +files_modified: [ + "config/backend.env", + "config/frontend.env", + "config/network.env", + "config/docker.env", + "config/README.md", + "backend/config_loader.py", + "backend/config_manager.py", + "backend/entrypoint.sh", + "deploy.sh", + "run_standalone.sh", + "export_prod.sh", + "install_service.sh", + "docker-compose.yml", + "DEPLOYMENT.md", + "README.md" +] +autonomous: true +--- + +# Phase 7: Config Consolidation - Implementation Plan + +**Objective:** Establish a centralized config/ folder structure, migrate all configurations from root level to config/, and update all scripts and code to use the new structure while maintaining backward compatibility. + +**Success Criteria:** +- Config folder exists with all required configuration files +- All scripts reference config folder instead of root-level env files +- Docker deployment works with new config structure +- Standalone deployment works with new config structure +- Backward compatibility: old inventory.env still loads if needed +- All documentation updated +- Root directory cleaned of unnecessary files + +--- + +## Task 1: Create Config Folder Structure + +**Read First:** +- Current project root layout (understand what we're migrating from) +- Current inventory.env and variants +- PROJECT_ARCHITECTURE.md (reference tech stack and requirements) + +**Action:** +1. Create `config/` folder in project root: `mkdir -p config` +2. Create `config/README.md` with documentation of all config files and their purposes +3. Ensure config/ is tracked in git (add to .gitignore if needed, or ensure it's not in .gitignore) + +**Acceptance Criteria:** +- `config/` directory exists in project root +- `config/README.md` exists and documents the purpose of each config file +- `config/` appears in git status (is tracked) + +--- + +## Task 2: Create backend.env Configuration File + +**Read First:** +- Current `inventory.env` content and structure +- `backend/config_loader.py` to understand what variables are expected +- `backend/config_manager.py` to understand all used environment variables +- `backend/main.py` to see what environment variables are loaded + +**Action:** +1. Read existing `inventory.env` file +2. Extract backend-specific environment variables (database, AI keys, auth settings, JWT secrets) +3. Create `config/backend.env` with all backend-specific variables from inventory.env +4. Include meaningful comments explaining each variable +5. Use same values as inventory.env to maintain current functionality +6. Ensure format matches python-dotenv expectations + +**Acceptance Criteria:** +- `config/backend.env` exists and contains all backend-specific variables +- File format is valid for python-dotenv (KEY=VALUE format with comments) +- Contains at least: JWT_SECRET_KEY, GEMINI_API_KEY, LDAP settings, database config +- All values match original inventory.env + +--- + +## Task 3: Create network.env Configuration File + +**Read First:** +- Current `inventory.env` file +- `run_standalone.sh` to see what network variables it uses +- `deploy.sh` to see what network variables it references +- `docker-compose.yml` to understand port and network configuration + +**Action:** +1. Extract network/deployment-specific variables from inventory.env (ports, server IPs, SSL config, CORS settings) +2. Create `config/network.env` with these variables +3. Include meaningful comments for each variable +4. Ensure variables match what deploy.sh and run_standalone.sh expect +5. Include default values for development + +**Acceptance Criteria:** +- `config/network.env` exists with network-specific variables +- Contains at least: BACKEND_PORT, BACKEND_SSL_PORT, FRONTEND_PORT, FRONTEND_SSL_PORT, SERVER_IP, SSL_ENABLED +- All values match original inventory.env +- Format is bash-sourceable (KEY=VALUE) + +--- + +## Task 4: Create docker.env Configuration File + +**Read First:** +- Current `docker-compose.yml` file +- `inventory.env` file for current values +- Dockerfile files (backend/Dockerfile, frontend/Dockerfile) + +**Action:** +1. Create `config/docker.env` with variables specifically for docker-compose +2. Include variables that docker-compose.yml references in its environment sections +3. These may overlap with network.env but are docker-compose specific +4. Add comments explaining docker-specific context +5. Include any build arguments and docker-specific settings + +**Acceptance Criteria:** +- `config/docker.env` exists +- Contains Docker-specific environment variables +- Format is valid for docker-compose (KEY=VALUE) +- All required docker-compose.yml variables are present + +--- + +## Task 5: Create frontend.env Configuration File + +**Read First:** +- `frontend/package.json` to see if environment variables are used +- `frontend/next.config.mjs` to understand what env vars are needed +- `frontend/entrypoint.sh` to see how frontend loads configuration +- Any frontend environment setup in current codebase + +**Action:** +1. Create `config/frontend.env` with frontend-specific variables +2. Include API endpoint configuration, feature flags, service worker settings, etc. +3. Add comments explaining each variable's purpose +4. If frontend doesn't currently use env files, create minimal defaults for future use +5. Ensure Next.js compatible format + +**Acceptance Criteria:** +- `config/frontend.env` exists +- Contains frontend-specific variables (API_BASE_URL, feature flags, etc.) +- Format is valid for frontend configuration +- Documented with clear comments + +--- + +## Task 6: Update backend/config_loader.py + +**Read First:** +- Current `backend/config_loader.py` implementation +- Current `backend/config_manager.py` implementation +- `backend/main.py` to see how config_loader is used +- Current load order and fallback logic + +**Action:** +1. Update `config_loader.py` to change config loading order: + - Priority 1: System environment variables (already set by Docker/deployment) + - Priority 2: `config/backend.env` (new centralized location) + - Priority 3: `inventory.env` (backward compatibility) + - Priority 4: `backend/.env` (legacy location) + - Priority 5: Hardcoded defaults +2. Update file paths to look in config/ folder first +3. Update log messages to indicate which config file is being loaded +4. Ensure backward compatibility: if config/backend.env doesn't exist, fall back to inventory.env +5. Test that load_dotenv() calls work correctly with new paths + +**Acceptance Criteria:** +- `config_loader.py` contains logic to load from `config/backend.env` first +- Falls back to `inventory.env` if `config/backend.env` not found +- Load order matches: env vars > config/backend.env > inventory.env > backend/.env +- Log messages indicate which config file was loaded +- All environment variables are still accessible to rest of backend +- File contains comment explaining the new config structure + +--- + +## Task 7: Update backend/config_manager.py + +**Read First:** +- Current `backend/config_manager.py` implementation +- Look for any file paths that hardcode inventory.env +- Understand how config updates are written back to disk + +**Action:** +1. Update file paths to use `config/backend.env` instead of root `inventory.env` +2. Ensure write operations go to `config/backend.env` +3. Update comments to reflect new path +4. Verify get_config_path() returns path to config/backend.env +5. Ensure file operations handle non-existent config/ folder gracefully + +**Acceptance Criteria:** +- `config_manager.py` references `config/backend.env` instead of `inventory.env` +- get_config_path() returns correct path to config/backend.env +- Config updates are written to config/backend.env +- Error handling works if config/ folder doesn't exist + +--- + +## Task 8: Update backend/entrypoint.sh + +**Read First:** +- Current `backend/entrypoint.sh` content +- How environment variables are sourced +- Docker ENTRYPOINT and CMD configuration + +**Action:** +1. Update entrypoint.sh to source from `config/backend.env` instead of root location +2. Update path references to point to /app/config/backend.env (inside Docker container) +3. Maintain backward compatibility: try config/backend.env first, fall back to inventory.env +4. Add logging to show which config was loaded +5. Ensure entrypoint handles missing config gracefully + +**Acceptance Criteria:** +- `entrypoint.sh` sources from `config/backend.env` +- Falls back to `inventory.env` if config/backend.env not found +- Inside Docker, path is /app/config/backend.env +- Script logs which config file was loaded +- Script doesn't fail if config files don't exist + +--- + +## Task 9: Update deploy.sh Script + +**Read First:** +- Current `deploy.sh` implementation +- How environment variables are currently sourced +- Lines that reference inventory.env + +**Action:** +1. Update script to load from `config/network.env` and `config/docker.env` instead of `inventory.env` +2. Change: `export $(grep -v '^#' "inventory.env" | xargs)` to `export $(grep -v '^#' "config/network.env" | xargs)` +3. Add fallback: if config/network.env doesn't exist, use inventory.env +4. Update docker-compose calls to use config/docker.env via environment variable sourcing +5. Add validation: check that config/ folder exists before sourcing +6. Add helpful error message if config files are missing + +**Acceptance Criteria:** +- `deploy.sh` sources from `config/network.env` instead of `inventory.env` +- Includes fallback to `inventory.env` if config/network.env not found +- Validates config folder exists with helpful error message +- Docker-compose gets correct environment variables from config files +- Script still functions with new structure + +--- + +## Task 10: Update run_standalone.sh Script + +**Read First:** +- Current `run_standalone.sh` implementation +- Lines that reference CONFIG_PATH or inventory.env +- How network configuration is loaded +- Backend and frontend startup logic + +**Action:** +1. Update CONFIG_PATH to point to `config/network.env` +2. Change line: `CONFIG_PATH="$(cd "$(dirname "$0")" && pwd)/inventory.env"` to reference config/network.env +3. Add fallback: if config/network.env not found, try inventory.env +4. Update comment to reflect new config location +5. Ensure backend environment loading also uses config/backend.env (via PYTHONPATH or direct sourcing) +6. Add logging showing which config files are being used + +**Acceptance Criteria:** +- `run_standalone.sh` loads from `config/network.env` +- Falls back to `inventory.env` if config files not found +- Backend loads from `config/backend.env` via config_loader.py +- Script logs which config files are loaded +- Standalone mode works with new config structure + +--- + +## Task 11: Update export_prod.sh Script + +**Read First:** +- Current `export_prod.sh` implementation +- How it uses environment variables +- What configuration it needs + +**Action:** +1. Identify all environment variables used by export_prod.sh +2. Update script to source from `config/backend.env` and `config/network.env` +3. Add fallback to inventory.env for backward compatibility +4. Update comments to reflect new config loading +5. Ensure export functionality works with new config structure + +**Acceptance Criteria:** +- `export_prod.sh` sources correct config files from config/ folder +- Falls back to `inventory.env` if needed +- All required environment variables are available to the script +- Script functions correctly with new configuration structure + +--- + +## Task 12: Update install_service.sh Script + +**Read First:** +- Current `install_service.sh` implementation +- How it references configuration +- What paths it sets in systemd service files + +**Action:** +1. Update script to reference config/ folder in environment file paths +2. Update systemd service file generation to point to config/backend.env +3. If service uses EnvironmentFile, ensure it points to config/backend.env or both config/backend.env and config/network.env +4. Add validation that config/ folder exists +5. Update comments to explain new config structure + +**Acceptance Criteria:** +- `install_service.sh` references config/backend.env in service configuration +- Systemd service file has correct EnvironmentFile paths +- Service can load all required environment variables +- Script validates config folder exists + +--- + +## Task 13: Update docker-compose.yml + +**Read First:** +- Current `docker-compose.yml` file +- All env_file and environment references +- How inventory.env is currently used + +**Action:** +1. Update env_file directives to reference config/docker.env instead of inventory.env +2. Update any hardcoded environment variable references to use config/ equivalents +3. For services using env_file: `env_file: config/docker.env` +4. Ensure Docker build arguments reference correct config location +5. Add validation or comment explaining config folder requirement +6. Test that docker-compose can still read all needed variables + +**Acceptance Criteria:** +- `docker-compose.yml` references `config/docker.env` in env_file +- All services get correct environment variables +- Docker-compose validates successfully +- Docker services can access all required configuration + +--- + +## Task 14: Update DEPLOYMENT.md + +**Read First:** +- Current `DEPLOYMENT.md` content +- Current documentation structure +- Instructions for setting up configuration + +**Action:** +1. Add new section explaining config folder structure and purpose +2. Document each config file (backend.env, frontend.env, network.env, docker.env) +3. Explain which variables go in each config file +4. Update deployment instructions to reference config/ instead of inventory.env +5. Document backward compatibility behavior (still reads inventory.env if config/ not found) +6. Add troubleshooting section for common config issues +7. Update any examples to use new config paths + +**Acceptance Criteria:** +- DEPLOYMENT.md has section explaining config folder structure +- All four config files are documented with their purpose +- Deployment instructions reference config/ folder +- Backward compatibility is explained +- Examples use new config paths + +--- + +## Task 15: Update README.md + +**Read First:** +- Current `README.md` content +- Setup/quickstart section + +**Action:** +1. Add or update configuration setup section +2. Explain that config/ folder is where all configuration lives +3. For quick start, show example of creating config/backend.env +4. Link to DEPLOYMENT.md for detailed configuration reference +5. Keep it concise - detailed docs go in DEPLOYMENT.md + +**Acceptance Criteria:** +- README.md mentions config/ folder +- Setup section references config/ not inventory.env +- Configuration setup is clear for new users + +--- + +## Task 16: Verify Backward Compatibility and Test All Deployment Methods + +**Read First:** +- Current DEPLOYMENT.md +- All scripts that were updated +- Docker Compose setup +- Standalone setup requirements + +**Action:** +1. Ensure all scripts still function if inventory.env exists but config/ doesn't (backward compatibility) +2. Test Docker deployment: `./deploy.sh production` + - Verify: services start, environment variables are loaded correctly + - Check logs: which config file was loaded +3. Test Standalone deployment: `./run_standalone.sh` + - Verify: backend and frontend start correctly + - Verify: all environment variables available + - Verify: correct ports from config/network.env +4. Test environment variable override: set env var on command line, verify it takes precedence +5. Verify config/backend.env is loaded by backend: grep logs for config path message + +**Acceptance Criteria:** +- Docker deployment works with config/ structure +- Standalone deployment works with config/ structure +- Backward compatibility works: scripts still read inventory.env if config/ doesn't exist +- Environment variable precedence works: system env > config files +- All tests pass +- Logs show correct config file was loaded + +--- + +## Task 17: Audit and Clean Up Root Directory + +**Read First:** +- All files in project root directory +- Understand what each script does +- Current .gitignore + +**Action:** +1. Review all *.sh scripts in root: deploy.sh, run_standalone.sh, export_prod.sh, install_service.sh, __push_ALL_to_remote.sh +2. For each script, determine: + - Is it still used? (check git history, comments, references) + - Can it be archived/removed? + - Does it need updating for config folder? +3. For scripts that are truly obsolete: + - Move to dev_docs/ARCHIVE_LOGS.md or note in comment + - Do NOT delete without understanding purpose +4. Review inventory.env* files: + - Are inventory.env.example and inventory.env.template still needed? + - Update .gitignore to exclude inventory.env (old) but track config/ structure +5. Create summary of what's obsolete and what's actively used + +**Acceptance Criteria:** +- Review completed for all root-level scripts +- Documented which scripts are obsolete (if any) +- Confirmed which scripts are still actively used +- .gitignore updated if needed +- Summary created of root directory cleanup + +--- + +## Task 18: Final Validation and Documentation + +**Read First:** +- Updated DEPLOYMENT.md +- Updated README.md +- Updated scripts +- config/README.md + +**Action:** +1. Create comprehensive test checklist: + - Docker deployment test + - Standalone deployment test + - Environment variable override test + - Backward compatibility test + - Config file format validation +2. Run all tests and document results +3. Verify all config files exist and are properly formatted +4. Verify all scripts source from correct locations +5. Verify backend loads from config/backend.env +6. Create deployment.md section summarizing changes +7. Add entry to dev_docs/PLAN.md documenting completion + +**Acceptance Criteria:** +- All deployment methods tested and working +- Config files exist and are properly formatted +- Documentation updated and clear +- No errors in script execution +- Environment variables load from config/ as expected +- Phase marked complete in documentation + +--- + +## Notes + +**Backward Compatibility:** +- All scripts include fallback logic: try config/ first, then fall back to inventory.env +- This allows gradual migration without breaking existing deployments +- Eventually, old inventory.env files can be deprecated after migration period + +**Configuration Priority (from highest to lowest):** +1. System environment variables (set by Docker, deployment platform) +2. config/backend.env (new centralized backend config) +3. inventory.env (legacy, for backward compatibility) +4. backend/.env (legacy backend-specific) +5. Hardcoded defaults in code + +**Docker Deployment Flow:** +1. docker-compose.yml loads config/docker.env +2. Services get environment variables from docker-compose.yml +3. Backend entrypoint sources config/backend.env for additional variables +4. System env vars override everything + +**Standalone Deployment Flow:** +1. run_standalone.sh sources config/network.env +2. Backend activation sources config/backend.env via config_loader.py +3. All environment variables available to both backend and frontend +