docs(phase-7): add configuration consolidation planning

- Phase 7: Config Consolidation - CONTEXT.md: Defines scope, decisions, and requirements - PLAN.md: 18-task implementation plan with detailed acceptance criteria - Centralize all configs into config/ folder - Update deploy scripts, backend loading, and documentation - Maintain backward compatibility with inventory.env Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-04-23 12:07:39 +03:00
parent 8aeabcf1f5
commit 2b7a2b3a89
2 changed files with 604 additions and 0 deletions
--- a/.planning/phases/07-config-consolidation/07-CONTEXT.md
+++ b/.planning/phases/07-config-consolidation/07-CONTEXT.md
@@ -0,0 +1,102 @@
+# Phase 7: Config Consolidation - Context
+
+**Gathered:** 2026-04-23
+**Status:** Ready for planning
+**Source:** User Requirements
+
+---
+
+## Phase Boundary
+
+Consolidate all application configuration files into a centralized `config/` folder in the project root. This includes backend configurations, frontend configurations, deployment settings, and network configurations. Update all deployment scripts, application startup procedures, and backend/frontend code to load configurations from this centralized location.
+
+**Scope:**
+- Create and establish `config/` folder as the single source of truth for all application configuration
+- Migrate existing configuration files (inventory.env and variants) to config folder with meaningful names
+- Update all scripts (deploy.sh, run_standalone.sh, etc.) to reference the new config location
+- Refactor backend config_loader.py and config_manager.py to read from config folder
+- Update frontend environment loading if applicable
+- Verify and clean up root directory scripts that are no longer needed
+- Ensure Docker deployment, standalone deployment, and development all work correctly with new structure
+
+**Deliverables:**
+- `config/` folder with structured configuration files
+- Updated backend configuration loading mechanism
+- Updated deployment scripts
+- Updated startup procedures (Docker and standalone)
+- Documentation of configuration structure in README/DEPLOYMENT.md
+
+---
+
+## Implementation Decisions
+
+### Configuration Structure
+- Create `config/` folder in project root as the central location for all configurations
+- Use meaningful filenames: `backend.env`, `frontend.env`, `network.env`, `docker.env` (for docker-compose)
+- Keep backward compatibility: Support both old (`inventory.env`) and new (`config/backend.env`) paths during transition
+- Environment variable precedence: System env vars > Docker build args > config/backend.env > defaults in code
+
+### Backend Loading
+- Update `backend/config_loader.py` to look for configs in `config/backend.env` first
+- If not found, fall back to `inventory.env` (backward compatibility)
+- If neither exists, use system environment variables
+- Log which config source is being used for debugging
+
+### Deployment Scripts
+- `deploy.sh`: Source environment from `config/network.env` or `config/docker.env` instead of `inventory.env`
+- `run_standalone.sh`: Source configuration from `config/backend.env` and `config/network.env`
+- Docker Compose: Updated to reference `config/docker.env`
+- Scripts should create symlinks or validate that config folder exists before running
+
+### Cleanup Actions
+- Remove or archive `inventory.env` templates after migration
+- Audit root directory scripts for redundancy
+- Update .gitignore to ensure config/ folder structure is tracked (if needed)
+
+### Documentation Updates
+- Update DEPLOYMENT.md with new config folder structure
+- Update README.md with configuration setup instructions
+- Add config/README.md explaining each config file's purpose
+
+---
+
+## Specific Ideas
+
+1. **Config Files to Create:**
+   - `config/backend.env` — Backend-specific variables (database, AI keys, auth settings)
+   - `config/frontend.env` — Frontend-specific variables (API endpoints, feature flags)
+   - `config/network.env` — Network/deployment variables (ports, SSL, server IPs)
+   - `config/docker.env` — Docker-specific overrides (for docker-compose.yml)
+   - `config/README.md` — Documentation of all configuration files
+
+2. **Scripts to Update:**
+   - `deploy.sh` — Update to source from config/ folder
+   - `run_standalone.sh` — Update network config loading
+   - `export_prod.sh` — Ensure it uses new config structure
+   - `install_service.sh` — Update systemd service to reference new config paths
+   - `__push_ALL_to_remote.sh` — Review for config-related updates
+
+3. **Backend Changes:**
+   - `backend/config_loader.py` — Update load order: config/backend.env > inventory.env > env vars
+   - `backend/config_manager.py` — Update to read/write from config/backend.env
+   - `backend/entrypoint.sh` — Source from config/ instead of root level
+
+4. **Testing Requirements:**
+   - Docker deployment with new config structure
+   - Standalone deployment with new config structure
+   - Environment variable override behavior still works
+   - Backward compatibility: old inventory.env still loads if config/ doesn't exist
+
+---
+
+## Deferred Ideas
+
+- Dynamic config hot-reload without restart (future optimization)
+- Config validation framework (future enhancement)
+- Encrypted sensitive values in config files (future security enhancement)
+- Web UI for configuration management (future feature)
+
+---
+
+*Phase: 7-config-consolidation*  
+*Context gathered: 2026-04-23 from user requirements*
--- a/.planning/phases/07-config-consolidation/07-PLAN.md
+++ b/.planning/phases/07-config-consolidation/07-PLAN.md
@@ -0,0 +1,502 @@
+---
+wave: 1
+depends_on: []
+files_modified: [
+  "config/backend.env",
+  "config/frontend.env", 
+  "config/network.env",
+  "config/docker.env",
+  "config/README.md",
+  "backend/config_loader.py",
+  "backend/config_manager.py",
+  "backend/entrypoint.sh",
+  "deploy.sh",
+  "run_standalone.sh",
+  "export_prod.sh",
+  "install_service.sh",
+  "docker-compose.yml",
+  "DEPLOYMENT.md",
+  "README.md"
+]
+autonomous: true
+---
+
+# Phase 7: Config Consolidation - Implementation Plan
+
+**Objective:** Establish a centralized config/ folder structure, migrate all configurations from root level to config/, and update all scripts and code to use the new structure while maintaining backward compatibility.
+
+**Success Criteria:**
+- Config folder exists with all required configuration files
+- All scripts reference config folder instead of root-level env files
+- Docker deployment works with new config structure
+- Standalone deployment works with new config structure
+- Backward compatibility: old inventory.env still loads if needed
+- All documentation updated
+- Root directory cleaned of unnecessary files
+
+---
+
+## Task 1: Create Config Folder Structure
+
+**Read First:**
+- Current project root layout (understand what we're migrating from)
+- Current inventory.env and variants
+- PROJECT_ARCHITECTURE.md (reference tech stack and requirements)
+
+**Action:**
+1. Create `config/` folder in project root: `mkdir -p config`
+2. Create `config/README.md` with documentation of all config files and their purposes
+3. Ensure config/ is tracked in git (add to .gitignore if needed, or ensure it's not in .gitignore)
+
+**Acceptance Criteria:**
+- `config/` directory exists in project root
+- `config/README.md` exists and documents the purpose of each config file
+- `config/` appears in git status (is tracked)
+
+---
+
+## Task 2: Create backend.env Configuration File
+
+**Read First:**
+- Current `inventory.env` content and structure
+- `backend/config_loader.py` to understand what variables are expected
+- `backend/config_manager.py` to understand all used environment variables
+- `backend/main.py` to see what environment variables are loaded
+
+**Action:**
+1. Read existing `inventory.env` file
+2. Extract backend-specific environment variables (database, AI keys, auth settings, JWT secrets)
+3. Create `config/backend.env` with all backend-specific variables from inventory.env
+4. Include meaningful comments explaining each variable
+5. Use same values as inventory.env to maintain current functionality
+6. Ensure format matches python-dotenv expectations
+
+**Acceptance Criteria:**
+- `config/backend.env` exists and contains all backend-specific variables
+- File format is valid for python-dotenv (KEY=VALUE format with comments)
+- Contains at least: JWT_SECRET_KEY, GEMINI_API_KEY, LDAP settings, database config
+- All values match original inventory.env
+
+---
+
+## Task 3: Create network.env Configuration File
+
+**Read First:**
+- Current `inventory.env` file
+- `run_standalone.sh` to see what network variables it uses
+- `deploy.sh` to see what network variables it references
+- `docker-compose.yml` to understand port and network configuration
+
+**Action:**
+1. Extract network/deployment-specific variables from inventory.env (ports, server IPs, SSL config, CORS settings)
+2. Create `config/network.env` with these variables
+3. Include meaningful comments for each variable
+4. Ensure variables match what deploy.sh and run_standalone.sh expect
+5. Include default values for development
+
+**Acceptance Criteria:**
+- `config/network.env` exists with network-specific variables
+- Contains at least: BACKEND_PORT, BACKEND_SSL_PORT, FRONTEND_PORT, FRONTEND_SSL_PORT, SERVER_IP, SSL_ENABLED
+- All values match original inventory.env
+- Format is bash-sourceable (KEY=VALUE)
+
+---
+
+## Task 4: Create docker.env Configuration File
+
+**Read First:**
+- Current `docker-compose.yml` file
+- `inventory.env` file for current values
+- Dockerfile files (backend/Dockerfile, frontend/Dockerfile)
+
+**Action:**
+1. Create `config/docker.env` with variables specifically for docker-compose
+2. Include variables that docker-compose.yml references in its environment sections
+3. These may overlap with network.env but are docker-compose specific
+4. Add comments explaining docker-specific context
+5. Include any build arguments and docker-specific settings
+
+**Acceptance Criteria:**
+- `config/docker.env` exists
+- Contains Docker-specific environment variables
+- Format is valid for docker-compose (KEY=VALUE)
+- All required docker-compose.yml variables are present
+
+---
+
+## Task 5: Create frontend.env Configuration File
+
+**Read First:**
+- `frontend/package.json` to see if environment variables are used
+- `frontend/next.config.mjs` to understand what env vars are needed
+- `frontend/entrypoint.sh` to see how frontend loads configuration
+- Any frontend environment setup in current codebase
+
+**Action:**
+1. Create `config/frontend.env` with frontend-specific variables
+2. Include API endpoint configuration, feature flags, service worker settings, etc.
+3. Add comments explaining each variable's purpose
+4. If frontend doesn't currently use env files, create minimal defaults for future use
+5. Ensure Next.js compatible format
+
+**Acceptance Criteria:**
+- `config/frontend.env` exists
+- Contains frontend-specific variables (API_BASE_URL, feature flags, etc.)
+- Format is valid for frontend configuration
+- Documented with clear comments
+
+---
+
+## Task 6: Update backend/config_loader.py
+
+**Read First:**
+- Current `backend/config_loader.py` implementation
+- Current `backend/config_manager.py` implementation
+- `backend/main.py` to see how config_loader is used
+- Current load order and fallback logic
+
+**Action:**
+1. Update `config_loader.py` to change config loading order:
+   - Priority 1: System environment variables (already set by Docker/deployment)
+   - Priority 2: `config/backend.env` (new centralized location)
+   - Priority 3: `inventory.env` (backward compatibility)
+   - Priority 4: `backend/.env` (legacy location)
+   - Priority 5: Hardcoded defaults
+2. Update file paths to look in config/ folder first
+3. Update log messages to indicate which config file is being loaded
+4. Ensure backward compatibility: if config/backend.env doesn't exist, fall back to inventory.env
+5. Test that load_dotenv() calls work correctly with new paths
+
+**Acceptance Criteria:**
+- `config_loader.py` contains logic to load from `config/backend.env` first
+- Falls back to `inventory.env` if `config/backend.env` not found
+- Load order matches: env vars > config/backend.env > inventory.env > backend/.env
+- Log messages indicate which config file was loaded
+- All environment variables are still accessible to rest of backend
+- File contains comment explaining the new config structure
+
+---
+
+## Task 7: Update backend/config_manager.py
+
+**Read First:**
+- Current `backend/config_manager.py` implementation
+- Look for any file paths that hardcode inventory.env
+- Understand how config updates are written back to disk
+
+**Action:**
+1. Update file paths to use `config/backend.env` instead of root `inventory.env`
+2. Ensure write operations go to `config/backend.env`
+3. Update comments to reflect new path
+4. Verify get_config_path() returns path to config/backend.env
+5. Ensure file operations handle non-existent config/ folder gracefully
+
+**Acceptance Criteria:**
+- `config_manager.py` references `config/backend.env` instead of `inventory.env`
+- get_config_path() returns correct path to config/backend.env
+- Config updates are written to config/backend.env
+- Error handling works if config/ folder doesn't exist
+
+---
+
+## Task 8: Update backend/entrypoint.sh
+
+**Read First:**
+- Current `backend/entrypoint.sh` content
+- How environment variables are sourced
+- Docker ENTRYPOINT and CMD configuration
+
+**Action:**
+1. Update entrypoint.sh to source from `config/backend.env` instead of root location
+2. Update path references to point to /app/config/backend.env (inside Docker container)
+3. Maintain backward compatibility: try config/backend.env first, fall back to inventory.env
+4. Add logging to show which config was loaded
+5. Ensure entrypoint handles missing config gracefully
+
+**Acceptance Criteria:**
+- `entrypoint.sh` sources from `config/backend.env`
+- Falls back to `inventory.env` if config/backend.env not found
+- Inside Docker, path is /app/config/backend.env
+- Script logs which config file was loaded
+- Script doesn't fail if config files don't exist
+
+---
+
+## Task 9: Update deploy.sh Script
+
+**Read First:**
+- Current `deploy.sh` implementation
+- How environment variables are currently sourced
+- Lines that reference inventory.env
+
+**Action:**
+1. Update script to load from `config/network.env` and `config/docker.env` instead of `inventory.env`
+2. Change: `export $(grep -v '^#' "inventory.env" | xargs)` to `export $(grep -v '^#' "config/network.env" | xargs)`
+3. Add fallback: if config/network.env doesn't exist, use inventory.env
+4. Update docker-compose calls to use config/docker.env via environment variable sourcing
+5. Add validation: check that config/ folder exists before sourcing
+6. Add helpful error message if config files are missing
+
+**Acceptance Criteria:**
+- `deploy.sh` sources from `config/network.env` instead of `inventory.env`
+- Includes fallback to `inventory.env` if config/network.env not found
+- Validates config folder exists with helpful error message
+- Docker-compose gets correct environment variables from config files
+- Script still functions with new structure
+
+---
+
+## Task 10: Update run_standalone.sh Script
+
+**Read First:**
+- Current `run_standalone.sh` implementation
+- Lines that reference CONFIG_PATH or inventory.env
+- How network configuration is loaded
+- Backend and frontend startup logic
+
+**Action:**
+1. Update CONFIG_PATH to point to `config/network.env`
+2. Change line: `CONFIG_PATH="$(cd "$(dirname "$0")" && pwd)/inventory.env"` to reference config/network.env
+3. Add fallback: if config/network.env not found, try inventory.env
+4. Update comment to reflect new config location
+5. Ensure backend environment loading also uses config/backend.env (via PYTHONPATH or direct sourcing)
+6. Add logging showing which config files are being used
+
+**Acceptance Criteria:**
+- `run_standalone.sh` loads from `config/network.env`
+- Falls back to `inventory.env` if config files not found
+- Backend loads from `config/backend.env` via config_loader.py
+- Script logs which config files are loaded
+- Standalone mode works with new config structure
+
+---
+
+## Task 11: Update export_prod.sh Script
+
+**Read First:**
+- Current `export_prod.sh` implementation
+- How it uses environment variables
+- What configuration it needs
+
+**Action:**
+1. Identify all environment variables used by export_prod.sh
+2. Update script to source from `config/backend.env` and `config/network.env`
+3. Add fallback to inventory.env for backward compatibility
+4. Update comments to reflect new config loading
+5. Ensure export functionality works with new config structure
+
+**Acceptance Criteria:**
+- `export_prod.sh` sources correct config files from config/ folder
+- Falls back to `inventory.env` if needed
+- All required environment variables are available to the script
+- Script functions correctly with new configuration structure
+
+---
+
+## Task 12: Update install_service.sh Script
+
+**Read First:**
+- Current `install_service.sh` implementation
+- How it references configuration
+- What paths it sets in systemd service files
+
+**Action:**
+1. Update script to reference config/ folder in environment file paths
+2. Update systemd service file generation to point to config/backend.env
+3. If service uses EnvironmentFile, ensure it points to config/backend.env or both config/backend.env and config/network.env
+4. Add validation that config/ folder exists
+5. Update comments to explain new config structure
+
+**Acceptance Criteria:**
+- `install_service.sh` references config/backend.env in service configuration
+- Systemd service file has correct EnvironmentFile paths
+- Service can load all required environment variables
+- Script validates config folder exists
+
+---
+
+## Task 13: Update docker-compose.yml
+
+**Read First:**
+- Current `docker-compose.yml` file
+- All env_file and environment references
+- How inventory.env is currently used
+
+**Action:**
+1. Update env_file directives to reference config/docker.env instead of inventory.env
+2. Update any hardcoded environment variable references to use config/ equivalents
+3. For services using env_file: `env_file: config/docker.env`
+4. Ensure Docker build arguments reference correct config location
+5. Add validation or comment explaining config folder requirement
+6. Test that docker-compose can still read all needed variables
+
+**Acceptance Criteria:**
+- `docker-compose.yml` references `config/docker.env` in env_file
+- All services get correct environment variables
+- Docker-compose validates successfully
+- Docker services can access all required configuration
+
+---
+
+## Task 14: Update DEPLOYMENT.md
+
+**Read First:**
+- Current `DEPLOYMENT.md` content
+- Current documentation structure
+- Instructions for setting up configuration
+
+**Action:**
+1. Add new section explaining config folder structure and purpose
+2. Document each config file (backend.env, frontend.env, network.env, docker.env)
+3. Explain which variables go in each config file
+4. Update deployment instructions to reference config/ instead of inventory.env
+5. Document backward compatibility behavior (still reads inventory.env if config/ not found)
+6. Add troubleshooting section for common config issues
+7. Update any examples to use new config paths
+
+**Acceptance Criteria:**
+- DEPLOYMENT.md has section explaining config folder structure
+- All four config files are documented with their purpose
+- Deployment instructions reference config/ folder
+- Backward compatibility is explained
+- Examples use new config paths
+
+---
+
+## Task 15: Update README.md
+
+**Read First:**
+- Current `README.md` content
+- Setup/quickstart section
+
+**Action:**
+1. Add or update configuration setup section
+2. Explain that config/ folder is where all configuration lives
+3. For quick start, show example of creating config/backend.env
+4. Link to DEPLOYMENT.md for detailed configuration reference
+5. Keep it concise - detailed docs go in DEPLOYMENT.md
+
+**Acceptance Criteria:**
+- README.md mentions config/ folder
+- Setup section references config/ not inventory.env
+- Configuration setup is clear for new users
+
+---
+
+## Task 16: Verify Backward Compatibility and Test All Deployment Methods
+
+**Read First:**
+- Current DEPLOYMENT.md
+- All scripts that were updated
+- Docker Compose setup
+- Standalone setup requirements
+
+**Action:**
+1. Ensure all scripts still function if inventory.env exists but config/ doesn't (backward compatibility)
+2. Test Docker deployment: `./deploy.sh production`
+   - Verify: services start, environment variables are loaded correctly
+   - Check logs: which config file was loaded
+3. Test Standalone deployment: `./run_standalone.sh`
+   - Verify: backend and frontend start correctly
+   - Verify: all environment variables available
+   - Verify: correct ports from config/network.env
+4. Test environment variable override: set env var on command line, verify it takes precedence
+5. Verify config/backend.env is loaded by backend: grep logs for config path message
+
+**Acceptance Criteria:**
+- Docker deployment works with config/ structure
+- Standalone deployment works with config/ structure
+- Backward compatibility works: scripts still read inventory.env if config/ doesn't exist
+- Environment variable precedence works: system env > config files
+- All tests pass
+- Logs show correct config file was loaded
+
+---
+
+## Task 17: Audit and Clean Up Root Directory
+
+**Read First:**
+- All files in project root directory
+- Understand what each script does
+- Current .gitignore
+
+**Action:**
+1. Review all *.sh scripts in root: deploy.sh, run_standalone.sh, export_prod.sh, install_service.sh, __push_ALL_to_remote.sh
+2. For each script, determine:
+   - Is it still used? (check git history, comments, references)
+   - Can it be archived/removed?
+   - Does it need updating for config folder?
+3. For scripts that are truly obsolete:
+   - Move to dev_docs/ARCHIVE_LOGS.md or note in comment
+   - Do NOT delete without understanding purpose
+4. Review inventory.env* files:
+   - Are inventory.env.example and inventory.env.template still needed?
+   - Update .gitignore to exclude inventory.env (old) but track config/ structure
+5. Create summary of what's obsolete and what's actively used
+
+**Acceptance Criteria:**
+- Review completed for all root-level scripts
+- Documented which scripts are obsolete (if any)
+- Confirmed which scripts are still actively used
+- .gitignore updated if needed
+- Summary created of root directory cleanup
+
+---
+
+## Task 18: Final Validation and Documentation
+
+**Read First:**
+- Updated DEPLOYMENT.md
+- Updated README.md
+- Updated scripts
+- config/README.md
+
+**Action:**
+1. Create comprehensive test checklist:
+   - Docker deployment test
+   - Standalone deployment test
+   - Environment variable override test
+   - Backward compatibility test
+   - Config file format validation
+2. Run all tests and document results
+3. Verify all config files exist and are properly formatted
+4. Verify all scripts source from correct locations
+5. Verify backend loads from config/backend.env
+6. Create deployment.md section summarizing changes
+7. Add entry to dev_docs/PLAN.md documenting completion
+
+**Acceptance Criteria:**
+- All deployment methods tested and working
+- Config files exist and are properly formatted
+- Documentation updated and clear
+- No errors in script execution
+- Environment variables load from config/ as expected
+- Phase marked complete in documentation
+
+---
+
+## Notes
+
+**Backward Compatibility:**
+- All scripts include fallback logic: try config/ first, then fall back to inventory.env
+- This allows gradual migration without breaking existing deployments
+- Eventually, old inventory.env files can be deprecated after migration period
+
+**Configuration Priority (from highest to lowest):**
+1. System environment variables (set by Docker, deployment platform)
+2. config/backend.env (new centralized backend config)
+3. inventory.env (legacy, for backward compatibility)
+4. backend/.env (legacy backend-specific)
+5. Hardcoded defaults in code
+
+**Docker Deployment Flow:**
+1. docker-compose.yml loads config/docker.env
+2. Services get environment variables from docker-compose.yml
+3. Backend entrypoint sources config/backend.env for additional variables
+4. System env vars override everything
+
+**Standalone Deployment Flow:**
+1. run_standalone.sh sources config/network.env
+2. Backend activation sources config/backend.env via config_loader.py
+3. All environment variables available to both backend and frontend
+