tfm_ainventory/.planning/phases/07-config-consolidation/07-PLAN.md

---
wave: 1
depends_on: []
files_modified: [
  "config/backend.env",
  "config/frontend.env", 
  "config/network.env",
  "config/docker.env",
  "config/README.md",
  "backend/config_loader.py",
  "backend/config_manager.py",
  "backend/entrypoint.sh",
  "deploy.sh",
  "run_standalone.sh",
  "export_prod.sh",
  "install_service.sh",
  "docker-compose.yml",
  "DEPLOYMENT.md",
  "README.md"
]
autonomous: true
---

# Phase 7: Config Consolidation - Implementation Plan

**Objective:** Establish a centralized config/ folder structure, migrate all configurations from root level to config/, and update all scripts and code to use the new structure while maintaining backward compatibility.

**Success Criteria:**
- Config folder exists with all required configuration files
- All scripts reference config folder instead of root-level env files
- Docker deployment works with new config structure
- Standalone deployment works with new config structure
- Backward compatibility: old inventory.env still loads if needed
- All documentation updated
- Root directory cleaned of unnecessary files

---

## Task 1: Create Config Folder Structure

**Read First:**
- Current project root layout (understand what we're migrating from)
- Current inventory.env and variants
- PROJECT_ARCHITECTURE.md (reference tech stack and requirements)

**Action:**
1. Create `config/` folder in project root: `mkdir -p config`
2. Create `config/README.md` with documentation of all config files and their purposes
3. Ensure config/ is tracked in git (add to .gitignore if needed, or ensure it's not in .gitignore)

**Acceptance Criteria:**
- `config/` directory exists in project root
- `config/README.md` exists and documents the purpose of each config file
- `config/` appears in git status (is tracked)

---

## Task 2: Create backend.env Configuration File

**Read First:**
- Current `inventory.env` content and structure
- `backend/config_loader.py` to understand what variables are expected
- `backend/config_manager.py` to understand all used environment variables
- `backend/main.py` to see what environment variables are loaded

**Action:**
1. Read existing `inventory.env` file
2. Extract backend-specific environment variables (database, AI keys, auth settings, JWT secrets)
3. Create `config/backend.env` with all backend-specific variables from inventory.env
4. Include meaningful comments explaining each variable
5. Use same values as inventory.env to maintain current functionality
6. Ensure format matches python-dotenv expectations

**Acceptance Criteria:**
- `config/backend.env` exists and contains all backend-specific variables
- File format is valid for python-dotenv (KEY=VALUE format with comments)
- Contains at least: JWT_SECRET_KEY, GEMINI_API_KEY, LDAP settings, database config
- All values match original inventory.env

---

## Task 3: Create network.env Configuration File

**Read First:**
- Current `inventory.env` file
- `run_standalone.sh` to see what network variables it uses
- `deploy.sh` to see what network variables it references
- `docker-compose.yml` to understand port and network configuration

**Action:**
1. Extract network/deployment-specific variables from inventory.env (ports, server IPs, SSL config, CORS settings)
2. Create `config/network.env` with these variables
3. Include meaningful comments for each variable
4. Ensure variables match what deploy.sh and run_standalone.sh expect
5. Include default values for development

**Acceptance Criteria:**
- `config/network.env` exists with network-specific variables
- Contains at least: BACKEND_PORT, BACKEND_SSL_PORT, FRONTEND_PORT, FRONTEND_SSL_PORT, SERVER_IP, SSL_ENABLED
- All values match original inventory.env
- Format is bash-sourceable (KEY=VALUE)

---

## Task 4: Create docker.env Configuration File

**Read First:**
- Current `docker-compose.yml` file
- `inventory.env` file for current values
- Dockerfile files (backend/Dockerfile, frontend/Dockerfile)

**Action:**
1. Create `config/docker.env` with variables specifically for docker-compose
2. Include variables that docker-compose.yml references in its environment sections
3. These may overlap with network.env but are docker-compose specific
4. Add comments explaining docker-specific context
5. Include any build arguments and docker-specific settings

**Acceptance Criteria:**
- `config/docker.env` exists
- Contains Docker-specific environment variables
- Format is valid for docker-compose (KEY=VALUE)
- All required docker-compose.yml variables are present

---

## Task 5: Create frontend.env Configuration File

**Read First:**
- `frontend/package.json` to see if environment variables are used
- `frontend/next.config.mjs` to understand what env vars are needed
- `frontend/entrypoint.sh` to see how frontend loads configuration
- Any frontend environment setup in current codebase

**Action:**
1. Create `config/frontend.env` with frontend-specific variables
2. Include API endpoint configuration, feature flags, service worker settings, etc.
3. Add comments explaining each variable's purpose
4. If frontend doesn't currently use env files, create minimal defaults for future use
5. Ensure Next.js compatible format

**Acceptance Criteria:**
- `config/frontend.env` exists
- Contains frontend-specific variables (API_BASE_URL, feature flags, etc.)
- Format is valid for frontend configuration
- Documented with clear comments

---

## Task 6: Update backend/config_loader.py

**Read First:**
- Current `backend/config_loader.py` implementation
- Current `backend/config_manager.py` implementation
- `backend/main.py` to see how config_loader is used
- Current load order and fallback logic

**Action:**
1. Update `config_loader.py` to change config loading order:
   - Priority 1: System environment variables (already set by Docker/deployment)
   - Priority 2: `config/backend.env` (new centralized location)
   - Priority 3: `inventory.env` (backward compatibility)
   - Priority 4: `backend/.env` (legacy location)
   - Priority 5: Hardcoded defaults
2. Update file paths to look in config/ folder first
3. Update log messages to indicate which config file is being loaded
4. Ensure backward compatibility: if config/backend.env doesn't exist, fall back to inventory.env
5. Test that load_dotenv() calls work correctly with new paths

**Acceptance Criteria:**
- `config_loader.py` contains logic to load from `config/backend.env` first
- Falls back to `inventory.env` if `config/backend.env` not found
- Load order matches: env vars > config/backend.env > inventory.env > backend/.env
- Log messages indicate which config file was loaded
- All environment variables are still accessible to rest of backend
- File contains comment explaining the new config structure

---

## Task 7: Update backend/config_manager.py

**Read First:**
- Current `backend/config_manager.py` implementation
- Look for any file paths that hardcode inventory.env
- Understand how config updates are written back to disk

**Action:**
1. Update file paths to use `config/backend.env` instead of root `inventory.env`
2. Ensure write operations go to `config/backend.env`
3. Update comments to reflect new path
4. Verify get_config_path() returns path to config/backend.env
5. Ensure file operations handle non-existent config/ folder gracefully

**Acceptance Criteria:**
- `config_manager.py` references `config/backend.env` instead of `inventory.env`
- get_config_path() returns correct path to config/backend.env
- Config updates are written to config/backend.env
- Error handling works if config/ folder doesn't exist

---

## Task 8: Update backend/entrypoint.sh

**Read First:**
- Current `backend/entrypoint.sh` content
- How environment variables are sourced
- Docker ENTRYPOINT and CMD configuration

**Action:**
1. Update entrypoint.sh to source from `config/backend.env` instead of root location
2. Update path references to point to /app/config/backend.env (inside Docker container)
3. Maintain backward compatibility: try config/backend.env first, fall back to inventory.env
4. Add logging to show which config was loaded
5. Ensure entrypoint handles missing config gracefully

**Acceptance Criteria:**
- `entrypoint.sh` sources from `config/backend.env`
- Falls back to `inventory.env` if config/backend.env not found
- Inside Docker, path is /app/config/backend.env
- Script logs which config file was loaded
- Script doesn't fail if config files don't exist

---

## Task 9: Update deploy.sh Script

**Read First:**
- Current `deploy.sh` implementation
- How environment variables are currently sourced
- Lines that reference inventory.env

**Action:**
1. Update script to load from `config/network.env` and `config/docker.env` instead of `inventory.env`
2. Change: `export $(grep -v '^#' "inventory.env" | xargs)` to `export $(grep -v '^#' "config/network.env" | xargs)`
3. Add fallback: if config/network.env doesn't exist, use inventory.env
4. Update docker-compose calls to use config/docker.env via environment variable sourcing
5. Add validation: check that config/ folder exists before sourcing
6. Add helpful error message if config files are missing

**Acceptance Criteria:**
- `deploy.sh` sources from `config/network.env` instead of `inventory.env`
- Includes fallback to `inventory.env` if config/network.env not found
- Validates config folder exists with helpful error message
- Docker-compose gets correct environment variables from config files
- Script still functions with new structure

---

## Task 10: Update run_standalone.sh Script

**Read First:**
- Current `run_standalone.sh` implementation
- Lines that reference CONFIG_PATH or inventory.env
- How network configuration is loaded
- Backend and frontend startup logic

**Action:**
1. Update CONFIG_PATH to point to `config/network.env`
2. Change line: `CONFIG_PATH="$(cd "$(dirname "$0")" && pwd)/inventory.env"` to reference config/network.env
3. Add fallback: if config/network.env not found, try inventory.env
4. Update comment to reflect new config location
5. Ensure backend environment loading also uses config/backend.env (via PYTHONPATH or direct sourcing)
6. Add logging showing which config files are being used

**Acceptance Criteria:**
- `run_standalone.sh` loads from `config/network.env`
- Falls back to `inventory.env` if config files not found
- Backend loads from `config/backend.env` via config_loader.py
- Script logs which config files are loaded
- Standalone mode works with new config structure

---

## Task 11: Update export_prod.sh Script

**Read First:**
- Current `export_prod.sh` implementation
- How it uses environment variables
- What configuration it needs

**Action:**
1. Identify all environment variables used by export_prod.sh
2. Update script to source from `config/backend.env` and `config/network.env`
3. Add fallback to inventory.env for backward compatibility
4. Update comments to reflect new config loading
5. Ensure export functionality works with new config structure

**Acceptance Criteria:**
- `export_prod.sh` sources correct config files from config/ folder
- Falls back to `inventory.env` if needed
- All required environment variables are available to the script
- Script functions correctly with new configuration structure

---

## Task 12: Update install_service.sh Script

**Read First:**
- Current `install_service.sh` implementation
- How it references configuration
- What paths it sets in systemd service files

**Action:**
1. Update script to reference config/ folder in environment file paths
2. Update systemd service file generation to point to config/backend.env
3. If service uses EnvironmentFile, ensure it points to config/backend.env or both config/backend.env and config/network.env
4. Add validation that config/ folder exists
5. Update comments to explain new config structure

**Acceptance Criteria:**
- `install_service.sh` references config/backend.env in service configuration
- Systemd service file has correct EnvironmentFile paths
- Service can load all required environment variables
- Script validates config folder exists

---

## Task 13: Update docker-compose.yml

**Read First:**
- Current `docker-compose.yml` file
- All env_file and environment references
- How inventory.env is currently used

**Action:**
1. Update env_file directives to reference config/docker.env instead of inventory.env
2. Update any hardcoded environment variable references to use config/ equivalents
3. For services using env_file: `env_file: config/docker.env`
4. Ensure Docker build arguments reference correct config location
5. Add validation or comment explaining config folder requirement
6. Test that docker-compose can still read all needed variables

**Acceptance Criteria:**
- `docker-compose.yml` references `config/docker.env` in env_file
- All services get correct environment variables
- Docker-compose validates successfully
- Docker services can access all required configuration

---

## Task 14: Update DEPLOYMENT.md

**Read First:**
- Current `DEPLOYMENT.md` content
- Current documentation structure
- Instructions for setting up configuration

**Action:**
1. Add new section explaining config folder structure and purpose
2. Document each config file (backend.env, frontend.env, network.env, docker.env)
3. Explain which variables go in each config file
4. Update deployment instructions to reference config/ instead of inventory.env
5. Document backward compatibility behavior (still reads inventory.env if config/ not found)
6. Add troubleshooting section for common config issues
7. Update any examples to use new config paths

**Acceptance Criteria:**
- DEPLOYMENT.md has section explaining config folder structure
- All four config files are documented with their purpose
- Deployment instructions reference config/ folder
- Backward compatibility is explained
- Examples use new config paths

---

## Task 15: Update README.md

**Read First:**
- Current `README.md` content
- Setup/quickstart section

**Action:**
1. Add or update configuration setup section
2. Explain that config/ folder is where all configuration lives
3. For quick start, show example of creating config/backend.env
4. Link to DEPLOYMENT.md for detailed configuration reference
5. Keep it concise - detailed docs go in DEPLOYMENT.md

**Acceptance Criteria:**
- README.md mentions config/ folder
- Setup section references config/ not inventory.env
- Configuration setup is clear for new users

---

## Task 16: Verify Backward Compatibility and Test All Deployment Methods

**Read First:**
- Current DEPLOYMENT.md
- All scripts that were updated
- Docker Compose setup
- Standalone setup requirements

**Action:**
1. Ensure all scripts still function if inventory.env exists but config/ doesn't (backward compatibility)
2. Test Docker deployment: `./deploy.sh production`
   - Verify: services start, environment variables are loaded correctly
   - Check logs: which config file was loaded
3. Test Standalone deployment: `./run_standalone.sh`
   - Verify: backend and frontend start correctly
   - Verify: all environment variables available
   - Verify: correct ports from config/network.env
4. Test environment variable override: set env var on command line, verify it takes precedence
5. Verify config/backend.env is loaded by backend: grep logs for config path message

**Acceptance Criteria:**
- Docker deployment works with config/ structure
- Standalone deployment works with config/ structure
- Backward compatibility works: scripts still read inventory.env if config/ doesn't exist
- Environment variable precedence works: system env > config files
- All tests pass
- Logs show correct config file was loaded

---

## Task 17: Audit and Clean Up Root Directory

**Read First:**
- All files in project root directory
- Understand what each script does
- Current .gitignore

**Action:**
1. Review all *.sh scripts in root: deploy.sh, run_standalone.sh, export_prod.sh, install_service.sh, __push_ALL_to_remote.sh
2. For each script, determine:
   - Is it still used? (check git history, comments, references)
   - Can it be archived/removed?
   - Does it need updating for config folder?
3. For scripts that are truly obsolete:
   - Move to dev_docs/ARCHIVE_LOGS.md or note in comment
   - Do NOT delete without understanding purpose
4. Review inventory.env* files:
   - Are inventory.env.example and inventory.env.template still needed?
   - Update .gitignore to exclude inventory.env (old) but track config/ structure
5. Create summary of what's obsolete and what's actively used

**Acceptance Criteria:**
- Review completed for all root-level scripts
- Documented which scripts are obsolete (if any)
- Confirmed which scripts are still actively used
- .gitignore updated if needed
- Summary created of root directory cleanup

---

## Task 18: Final Validation and Documentation

**Read First:**
- Updated DEPLOYMENT.md
- Updated README.md
- Updated scripts
- config/README.md

**Action:**
1. Create comprehensive test checklist:
   - Docker deployment test
   - Standalone deployment test
   - Environment variable override test
   - Backward compatibility test
   - Config file format validation
2. Run all tests and document results
3. Verify all config files exist and are properly formatted
4. Verify all scripts source from correct locations
5. Verify backend loads from config/backend.env
6. Create deployment.md section summarizing changes
7. Add entry to dev_docs/PLAN.md documenting completion

**Acceptance Criteria:**
- All deployment methods tested and working
- Config files exist and are properly formatted
- Documentation updated and clear
- No errors in script execution
- Environment variables load from config/ as expected
- Phase marked complete in documentation

---

## Notes

**Backward Compatibility:**
- All scripts include fallback logic: try config/ first, then fall back to inventory.env
- This allows gradual migration without breaking existing deployments
- Eventually, old inventory.env files can be deprecated after migration period

**Configuration Priority (from highest to lowest):**
1. System environment variables (set by Docker, deployment platform)
2. config/backend.env (new centralized backend config)
3. inventory.env (legacy, for backward compatibility)
4. backend/.env (legacy backend-specific)
5. Hardcoded defaults in code

**Docker Deployment Flow:**
1. docker-compose.yml loads config/docker.env
2. Services get environment variables from docker-compose.yml
3. Backend entrypoint sources config/backend.env for additional variables
4. System env vars override everything

**Standalone Deployment Flow:**
1. run_standalone.sh sources config/network.env
2. Backend activation sources config/backend.env via config_loader.py
3. All environment variables available to both backend and frontend