tfm_ainventory/.planning/phases/07-config-consolidation/07-PLAN.md

wave, depends_on, files_modified, autonomous

wave	depends_on	files_modified	autonomous
1		config/backend.env config/frontend.env config/network.env config/docker.env config/README.md backend/config_loader.py backend/config_manager.py backend/entrypoint.sh deploy.sh run_standalone.sh export_prod.sh install_service.sh docker-compose.yml DEPLOYMENT.md README.md	true

Phase 7: Config Consolidation - Implementation Plan

Objective: Establish a centralized config/ folder structure, migrate all configurations from root level to config/, and update all scripts and code to use the new structure while maintaining backward compatibility.

Success Criteria:

• Config folder exists with all required configuration files
• All scripts reference config folder instead of root-level env files
• Docker deployment works with new config structure
• Standalone deployment works with new config structure
• Backward compatibility: old inventory.env still loads if needed
• All documentation updated
• Root directory cleaned of unnecessary files

---

Task 1: Create Config Folder Structure

Read First:

• Current project root layout (understand what we're migrating from)
• Current inventory.env and variants
• PROJECT_ARCHITECTURE.md (reference tech stack and requirements)

Action:

1. Create config/ folder in project root: mkdir -p config
2. Create config/README.md with documentation of all config files and their purposes
3. Ensure config/ is tracked in git (add to .gitignore if needed, or ensure it's not in .gitignore)

Acceptance Criteria:

• config/ directory exists in project root
• config/README.md exists and documents the purpose of each config file
• config/ appears in git status (is tracked)

---

Task 2: Create backend.env Configuration File

Read First:

• Current inventory.env content and structure
• backend/config_loader.py to understand what variables are expected
• backend/config_manager.py to understand all used environment variables
• backend/main.py to see what environment variables are loaded

Action:

1. Read existing inventory.env file
2. Extract backend-specific environment variables (database, AI keys, auth settings, JWT secrets)
3. Create config/backend.env with all backend-specific variables from inventory.env
4. Include meaningful comments explaining each variable
5. Use same values as inventory.env to maintain current functionality
6. Ensure format matches python-dotenv expectations

Acceptance Criteria:

• config/backend.env exists and contains all backend-specific variables
• File format is valid for python-dotenv (KEY=VALUE format with comments)
• Contains at least: JWT_SECRET_KEY, GEMINI_API_KEY, LDAP settings, database config
• All values match original inventory.env

---

Task 3: Create network.env Configuration File

Read First:

• Current inventory.env file
• run_standalone.sh to see what network variables it uses
• deploy.sh to see what network variables it references
• docker-compose.yml to understand port and network configuration

Action:

1. Extract network/deployment-specific variables from inventory.env (ports, server IPs, SSL config, CORS settings)
2. Create config/network.env with these variables
3. Include meaningful comments for each variable
4. Ensure variables match what deploy.sh and run_standalone.sh expect
5. Include default values for development

Acceptance Criteria:

• config/network.env exists with network-specific variables
• Contains at least: BACKEND_PORT, BACKEND_SSL_PORT, FRONTEND_PORT, FRONTEND_SSL_PORT, SERVER_IP, SSL_ENABLED
• All values match original inventory.env
• Format is bash-sourceable (KEY=VALUE)

---

Task 4: Create docker.env Configuration File

Read First:

• Current docker-compose.yml file
• inventory.env file for current values
• Dockerfile files (backend/Dockerfile, frontend/Dockerfile)

Action:

1. Create config/docker.env with variables specifically for docker-compose
2. Include variables that docker-compose.yml references in its environment sections
3. These may overlap with network.env but are docker-compose specific
4. Add comments explaining docker-specific context
5. Include any build arguments and docker-specific settings

Acceptance Criteria:

• config/docker.env exists
• Contains Docker-specific environment variables
• Format is valid for docker-compose (KEY=VALUE)
• All required docker-compose.yml variables are present

---

Task 5: Create frontend.env Configuration File

Read First:

• frontend/package.json to see if environment variables are used
• frontend/next.config.mjs to understand what env vars are needed
• frontend/entrypoint.sh to see how frontend loads configuration
• Any frontend environment setup in current codebase

Action:

1. Create config/frontend.env with frontend-specific variables
2. Include API endpoint configuration, feature flags, service worker settings, etc.
3. Add comments explaining each variable's purpose
4. If frontend doesn't currently use env files, create minimal defaults for future use
5. Ensure Next.js compatible format

Acceptance Criteria:

• config/frontend.env exists
• Contains frontend-specific variables (API_BASE_URL, feature flags, etc.)
• Format is valid for frontend configuration
• Documented with clear comments

---

Task 6: Update backend/config_loader.py

Read First:

• Current backend/config_loader.py implementation
• Current backend/config_manager.py implementation
• backend/main.py to see how config_loader is used
• Current load order and fallback logic

Action:

1. Update config_loader.py to change config loading order:
   • Priority 1: System environment variables (already set by Docker/deployment)
   • Priority 2: config/backend.env (new centralized location)
   • Priority 3: inventory.env (backward compatibility)
   • Priority 4: backend/.env (legacy location)
   • Priority 5: Hardcoded defaults
2. Update file paths to look in config/ folder first
3. Update log messages to indicate which config file is being loaded
4. Ensure backward compatibility: if config/backend.env doesn't exist, fall back to inventory.env
5. Test that load_dotenv() calls work correctly with new paths

Acceptance Criteria:

• config_loader.py contains logic to load from config/backend.env first
• Falls back to inventory.env if config/backend.env not found
• Load order matches: env vars > config/backend.env > inventory.env > backend/.env
• Log messages indicate which config file was loaded
• All environment variables are still accessible to rest of backend
• File contains comment explaining the new config structure

---

Task 7: Update backend/config_manager.py

Read First:

• Current backend/config_manager.py implementation
• Look for any file paths that hardcode inventory.env
• Understand how config updates are written back to disk

Action:

1. Update file paths to use config/backend.env instead of root inventory.env
2. Ensure write operations go to config/backend.env
3. Update comments to reflect new path
4. Verify get_config_path() returns path to config/backend.env
5. Ensure file operations handle non-existent config/ folder gracefully

Acceptance Criteria:

• config_manager.py references config/backend.env instead of inventory.env
• get_config_path() returns correct path to config/backend.env
• Config updates are written to config/backend.env
• Error handling works if config/ folder doesn't exist

---

Task 8: Update backend/entrypoint.sh

Read First:

• Current backend/entrypoint.sh content
• How environment variables are sourced
• Docker ENTRYPOINT and CMD configuration

Action:

1. Update entrypoint.sh to source from config/backend.env instead of root location
2. Update path references to point to /app/config/backend.env (inside Docker container)
3. Maintain backward compatibility: try config/backend.env first, fall back to inventory.env
4. Add logging to show which config was loaded
5. Ensure entrypoint handles missing config gracefully

Acceptance Criteria:

• entrypoint.sh sources from config/backend.env
• Falls back to inventory.env if config/backend.env not found
• Inside Docker, path is /app/config/backend.env
• Script logs which config file was loaded
• Script doesn't fail if config files don't exist

---

Task 9: Update deploy.sh Script

Read First:

• Current deploy.sh implementation
• How environment variables are currently sourced
• Lines that reference inventory.env

Action:

1. Update script to load from config/network.env and config/docker.env instead of inventory.env
2. Change: export $(grep -v '^#' "inventory.env" | xargs) to export $(grep -v '^#' "config/network.env" | xargs)
3. Add fallback: if config/network.env doesn't exist, use inventory.env
4. Update docker-compose calls to use config/docker.env via environment variable sourcing
5. Add validation: check that config/ folder exists before sourcing
6. Add helpful error message if config files are missing

Acceptance Criteria:

• deploy.sh sources from config/network.env instead of inventory.env
• Includes fallback to inventory.env if config/network.env not found
• Validates config folder exists with helpful error message
• Docker-compose gets correct environment variables from config files
• Script still functions with new structure

---

Task 10: Update run_standalone.sh Script

Read First:

• Current run_standalone.sh implementation
• Lines that reference CONFIG_PATH or inventory.env
• How network configuration is loaded
• Backend and frontend startup logic

Action:

1. Update CONFIG_PATH to point to config/network.env
2. Change line: CONFIG_PATH="$(cd "$(dirname "$0")" && pwd)/inventory.env" to reference config/network.env
3. Add fallback: if config/network.env not found, try inventory.env
4. Update comment to reflect new config location
5. Ensure backend environment loading also uses config/backend.env (via PYTHONPATH or direct sourcing)
6. Add logging showing which config files are being used

Acceptance Criteria:

• run_standalone.sh loads from config/network.env
• Falls back to inventory.env if config files not found
• Backend loads from config/backend.env via config_loader.py
• Script logs which config files are loaded
• Standalone mode works with new config structure

---

Task 11: Update export_prod.sh Script

Read First:

• Current export_prod.sh implementation
• How it uses environment variables
• What configuration it needs

Action:

1. Identify all environment variables used by export_prod.sh
2. Update script to source from config/backend.env and config/network.env
3. Add fallback to inventory.env for backward compatibility
4. Update comments to reflect new config loading
5. Ensure export functionality works with new config structure

Acceptance Criteria:

• export_prod.sh sources correct config files from config/ folder
• Falls back to inventory.env if needed
• All required environment variables are available to the script
• Script functions correctly with new configuration structure

---

Task 12: Update install_service.sh Script

Read First:

• Current install_service.sh implementation
• How it references configuration
• What paths it sets in systemd service files

Action:

1. Update script to reference config/ folder in environment file paths
2. Update systemd service file generation to point to config/backend.env
3. If service uses EnvironmentFile, ensure it points to config/backend.env or both config/backend.env and config/network.env
4. Add validation that config/ folder exists
5. Update comments to explain new config structure

Acceptance Criteria:

• install_service.sh references config/backend.env in service configuration
• Systemd service file has correct EnvironmentFile paths
• Service can load all required environment variables
• Script validates config folder exists

---

Task 13: Update docker-compose.yml

Read First:

• Current docker-compose.yml file
• All env_file and environment references
• How inventory.env is currently used

Action:

1. Update env_file directives to reference config/docker.env instead of inventory.env
2. Update any hardcoded environment variable references to use config/ equivalents
3. For services using env_file: env_file: config/docker.env
4. Ensure Docker build arguments reference correct config location
5. Add validation or comment explaining config folder requirement
6. Test that docker-compose can still read all needed variables

Acceptance Criteria:

• docker-compose.yml references config/docker.env in env_file
• All services get correct environment variables
• Docker-compose validates successfully
• Docker services can access all required configuration

---

Task 14: Update DEPLOYMENT.md

Read First:

• Current DEPLOYMENT.md content
• Current documentation structure
• Instructions for setting up configuration

Action:

1. Add new section explaining config folder structure and purpose
2. Document each config file (backend.env, frontend.env, network.env, docker.env)
3. Explain which variables go in each config file
4. Update deployment instructions to reference config/ instead of inventory.env
5. Document backward compatibility behavior (still reads inventory.env if config/ not found)
6. Add troubleshooting section for common config issues
7. Update any examples to use new config paths

Acceptance Criteria:

• DEPLOYMENT.md has section explaining config folder structure
• All four config files are documented with their purpose
• Deployment instructions reference config/ folder
• Backward compatibility is explained
• Examples use new config paths

---

Task 15: Update README.md

Read First:

• Current README.md content
• Setup/quickstart section

Action:

1. Add or update configuration setup section
2. Explain that config/ folder is where all configuration lives
3. For quick start, show example of creating config/backend.env
4. Link to DEPLOYMENT.md for detailed configuration reference
5. Keep it concise - detailed docs go in DEPLOYMENT.md

Acceptance Criteria:

• README.md mentions config/ folder
• Setup section references config/ not inventory.env
• Configuration setup is clear for new users

---

Task 16: Verify Backward Compatibility and Test All Deployment Methods

Read First:

• Current DEPLOYMENT.md
• All scripts that were updated
• Docker Compose setup
• Standalone setup requirements

Action:

1. Ensure all scripts still function if inventory.env exists but config/ doesn't (backward compatibility)
2. Test Docker deployment: ./deploy.sh production
   • Verify: services start, environment variables are loaded correctly
   • Check logs: which config file was loaded
3. Test Standalone deployment: ./run_standalone.sh
   • Verify: backend and frontend start correctly
   • Verify: all environment variables available
   • Verify: correct ports from config/network.env
4. Test environment variable override: set env var on command line, verify it takes precedence
5. Verify config/backend.env is loaded by backend: grep logs for config path message

Acceptance Criteria:

• Docker deployment works with config/ structure
• Standalone deployment works with config/ structure
• Backward compatibility works: scripts still read inventory.env if config/ doesn't exist
• Environment variable precedence works: system env > config files
• All tests pass
• Logs show correct config file was loaded

---

Task 17: Audit and Clean Up Root Directory

Read First:

• All files in project root directory
• Understand what each script does
• Current .gitignore

Action:

1. Review all *.sh scripts in root: deploy.sh, run_standalone.sh, export_prod.sh, install_service.sh, __push_ALL_to_remote.sh
2. For each script, determine:
   • Is it still used? (check git history, comments, references)
   • Can it be archived/removed?
   • Does it need updating for config folder?
3. For scripts that are truly obsolete:
   • Move to dev_docs/ARCHIVE_LOGS.md or note in comment
   • Do NOT delete without understanding purpose
4. Review inventory.env* files:
   • Are inventory.env.example and inventory.env.template still needed?
   • Update .gitignore to exclude inventory.env (old) but track config/ structure
5. Create summary of what's obsolete and what's actively used

Acceptance Criteria:

• Review completed for all root-level scripts
• Documented which scripts are obsolete (if any)
• Confirmed which scripts are still actively used
• .gitignore updated if needed
• Summary created of root directory cleanup

---

Task 18: Final Validation and Documentation

Read First:

• Updated DEPLOYMENT.md
• Updated README.md
• Updated scripts
• config/README.md

Action:

1. Create comprehensive test checklist:
   • Docker deployment test
   • Standalone deployment test
   • Environment variable override test
   • Backward compatibility test
   • Config file format validation
2. Run all tests and document results
3. Verify all config files exist and are properly formatted
4. Verify all scripts source from correct locations
5. Verify backend loads from config/backend.env
6. Create deployment.md section summarizing changes
7. Add entry to dev_docs/PLAN.md documenting completion

Acceptance Criteria:

• All deployment methods tested and working
• Config files exist and are properly formatted
• Documentation updated and clear
• No errors in script execution
• Environment variables load from config/ as expected
• Phase marked complete in documentation

---

Notes

Backward Compatibility:

• All scripts include fallback logic: try config/ first, then fall back to inventory.env
• This allows gradual migration without breaking existing deployments
• Eventually, old inventory.env files can be deprecated after migration period

Configuration Priority (from highest to lowest):

1. System environment variables (set by Docker, deployment platform)
2. config/backend.env (new centralized backend config)
3. inventory.env (legacy, for backward compatibility)
4. backend/.env (legacy backend-specific)
5. Hardcoded defaults in code

Docker Deployment Flow:

1. docker-compose.yml loads config/docker.env
2. Services get environment variables from docker-compose.yml
3. Backend entrypoint sources config/backend.env for additional variables
4. System env vars override everything

Standalone Deployment Flow:

1. run_standalone.sh sources config/network.env
2. Backend activation sources config/backend.env via config_loader.py
3. All environment variables available to both backend and frontend