bede/tfm_ainventory
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
f2fb81024e7736cf032a959ae5d1d7773d08c6b1
tfm_ainventory/.planning/phases/07-config-consolidation/07-03-PLAN.md

584 lines
25 KiB
Markdown
Raw Blame History

---
phase: 07-config-consolidation
plan: 03
type: execute
wave: 2
depends_on:
- 07-01
files_modified:
- scripts/deploy.py
- scripts/run_standalone.py
- scripts/install_service.py
- scripts/export_prod.py
autonomous: true
requirements:
- PHASE-7-PYTHON-SCRIPTS
- PHASE-7-YAML-PARSING
- PHASE-7-DEPLOYMENT
user_setup: []
must_haves:
truths:
- "Python deployment scripts (deploy.py, run_standalone.py, install_service.py, export_prod.py) exist and parse YAML config"
- "All scripts parse config/*.yaml files using PyYAML (per D-05)"
- "deploy.py handles Docker deployment with health checks"
- "run_standalone.py launches backend and frontend without Docker"
- "install_service.py installs systemd service with new config paths"
- "export_prod.py exports production data/config for backups"
- "All scripts are executable and tested"
artifacts:
- path: "scripts/deploy.py"
provides: "Docker deployment with YAML config parsing, pre-flight checks, health validation"
exports: ["main()"]
min_lines: 150
- path: "scripts/run_standalone.py"
provides: "Standalone launcher for backend (FastAPI) and frontend (Next.js) with YAML config"
exports: ["main()"]
min_lines: 120
- path: "scripts/install_service.py"
provides: "Systemd service installation with config paths"
exports: ["main()"]
min_lines: 100
- path: "scripts/export_prod.py"
provides: "Production export/backup script with YAML config support"
exports: ["main()"]
min_lines: 100
key_links:
- from: "scripts/deploy.py"
to: "config/docker.yaml"
via: "PyYAML parsing for container config"
pattern: "yaml\\.safe_load.*docker\\.yaml"
- from: "scripts/run_standalone.py"
to: "config/backend.yaml"
via: "Read port and path config"
pattern: "yaml\\.safe_load.*backend\\.yaml"
- from: "scripts/install_service.py"
to: "inventory.service.template"
via: "Service file generation"
pattern: "template|service"
---
<objective>
Convert bash deployment scripts (deploy.sh, run_standalone.sh, install_service.sh, export_prod.sh) to Python with YAML config parsing. Provide consistent, maintainable deployment tooling that understands the new config structure.
Purpose: Implement D-05 (Python scripts with YAML parsing) for modern deployment infrastructure.
Output: 4 Python scripts in scripts/ folder with full deployment functionality and YAML config support.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/07-config-consolidation/07-CONTEXT.md
@PROJECT_ARCHITECTURE.md
@DEPLOYMENT.md
@deploy.sh
@run_standalone.sh
@install_service.sh
@export_prod.sh
@config/backend.yaml.example
@config/docker.yaml.example
@docker-compose.yml
</context>
<tasks>
<task type="auto">
<name>Task 1: Create scripts/deploy.py (Docker deployment with YAML config)</name>
<files>
scripts/deploy.py
</files>
<read_first>
- deploy.sh (current bash implementation to port)
- docker-compose.yml (structure and environment variables)
- config/docker.yaml.example (schema)
- config/backend.yaml.example (config structure)
- DEPLOYMENT.md (deployment procedure documentation)
</read_first>
<action>
Create scripts/deploy.py to replace deploy.sh with Python implementation (per D-05).
**Key features:**
1. **Script signature and usage:**
```bash
python3 scripts/deploy.py [production|staging|development] [--rebuild]
```
2. **Core functionality:**
- Pre-flight checks: Docker, Docker Compose, docker-compose.yml, config files
- Load config from config/docker.yaml and config/network.yaml
- Port availability checks (from network.yaml: backend_port, frontend_port, etc.)
- Environment file validation (config/backend.yaml exists and has required values)
- Docker Compose up with appropriate flags (rebuild if --rebuild)
- Health check polling (curl to /health endpoints)
- Deployment report (services running, ports, access URLs)
3. **Config file parsing:**
- Use PyYAML to load config/docker.yaml (for container resource limits, image names)
- Use PyYAML to load config/network.yaml (for port numbers and SSL settings)
- Use PyYAML to load config/backend.yaml (to validate required values)
- Fallback to sensible defaults if config files missing (but log warnings)
4. **Pre-flight checks (Step 1-5):**
- [ ] docker command available
- [ ] docker-compose command available
- [ ] docker-compose.yml exists
- [ ] config/backend.yaml exists (with helpful error if missing)
- [ ] config/network.yaml exists (with helpful error if missing)
5. **Port availability check (Step 6):**
- Read backend_port, frontend_port, backend_ssl_port, frontend_ssl_port from network.yaml
- Use netstat or ss to check if ports are in use
- Error if ports occupied, suggest alternatives
6. **Environment validation (Step 7):**
- Check config/backend.yaml for required values: JWT_SECRET_KEY, primary_ai_provider
- Warn if API keys are placeholders
- Proceed with warning (not error) for optional values
7. **Docker Compose deployment (Step 8-9):**
- Run `docker-compose up -d` (or with --build if --rebuild flag)
- Capture and display output with color codes
- Catch errors and provide helpful debugging steps
8. **Health checks (Step 10-11):**
- Poll backend health: `curl http://localhost:{backend_port}/health` (retry logic)
- Poll frontend health: `curl http://localhost:{frontend_port}/` (retry logic)
- Wait up to 2 minutes for services to become healthy
- Display health status to user
9. **Deployment report (Step 12):**
- Display service status: `docker-compose ps`
- Display access URLs:
- Frontend: http://localhost:{frontend_port}
- Backend API: http://localhost:{backend_port}/docs
- HTTPS: https://localhost:{frontend_ssl_port} (if SSL enabled in network.yaml)
- Display next steps (logs, troubleshooting, etc.)
10. **Error handling:**
- Descriptive error messages with suggested fixes
- Log all actions and results to stdout/stderr
- Use color output (GREEN for success, RED for errors, YELLOW for warnings, BLUE for info)
- Exit codes: 0 for success, 1 for fatal error
11. **Logging:**
- Use Python logging module (not print)
- Log level: INFO by default, DEBUG if --verbose flag
- Each step logged: "Step N/M: Description..."
- Results logged at end: "Deployment complete, services healthy"
12. **Required libraries:**
- sys, os, subprocess, time, socket (built-in)
- yaml (PyYAML)
- argparse (for CLI args)
- logging (for logging)
- No external deployment libraries (keep it simple)
13. **Make executable:** `chmod +x scripts/deploy.py` and include shebang: `#!/usr/bin/env python3`
</action>
<verify>
- `test -f scripts/deploy.py && head -1 scripts/deploy.py | grep -q "python3"` (shebang present)
- `test -x scripts/deploy.py` (executable)
- `python3 -m py_compile scripts/deploy.py` (valid Python syntax)
- `python3 scripts/deploy.py --help | grep -q "deployment"` (help works)
- `grep -q "import yaml" scripts/deploy.py` (PyYAML imported)
- `grep -q "config/docker.yaml\|config/network.yaml" scripts/deploy.py` (loads config files)
- `grep -q "docker-compose" scripts/deploy.py` (calls docker-compose)
- `grep -q "curl.*health" scripts/deploy.py` (health checks present)
</verify>
<done>
scripts/deploy.py created with Docker deployment, YAML config parsing, health checks, and error handling.
</done>
</task>
<task type="auto">
<name>Task 2: Create scripts/run_standalone.py (Standalone launcher with YAML config)</name>
<files>
scripts/run_standalone.py
</files>
<read_first>
- run_standalone.sh (current bash implementation to port)
- config/backend.yaml.example (schema)
- config/frontend.yaml.example (schema)
- backend/main.py (backend entry point)
- frontend package.json or next.config.js (frontend startup)
</read_first>
<action>
Create scripts/run_standalone.py to replace run_standalone.sh with Python implementation (per D-05).
**Key features:**
1. **Script signature:**
```bash
python3 scripts/run_standalone.py [--backend-only|--frontend-only]
```
2. **Core functionality:**
- Load config from config/backend.yaml and config/frontend.yaml
- Start FastAPI backend (uvicorn)
- Start Next.js frontend (npm run dev or node server.js)
- Display console output from both processes
- Handle shutdown gracefully (SIGTERM/SIGINT kills both services)
- Display health status and access URLs
3. **Config file parsing:**
- Load config/backend.yaml to get: backend_port, data_dir, logs_dir, log_level
- Load config/frontend.yaml to get: frontend_port, backend_url
- Use defaults if config files missing (with warnings)
4. **Backend startup (--backend-only or default):**
- Command: `uvicorn backend.main:app --host 0.0.0.0 --port {backend_port} --reload`
- Set environment: DATA_DIR, LOGS_DIR, LOG_LEVEL (from config)
- Capture output and display with [BACKEND] prefix
- Wait for backend to log "Uvicorn running on..." or similar
- Verify backend is listening on backend_port
5. **Frontend startup (--frontend-only or default):**
- Command: `npm run dev` (if in development) or `node server.js` (if built)
- Set environment: NEXT_PUBLIC_API_URL (from config:frontend:backend_url)
- Capture output and display with [FRONTEND] prefix
- Wait for frontend to log "ready - started server on..." or similar
- Verify frontend is listening on frontend_port
6. **Process management:**
- Use subprocess.Popen with shell=False (for security)
- Manage both processes in list/tuple
- Handle SIGTERM/SIGINT (Ctrl+C) to kill both processes
- Display "Shutting down..." and wait for clean shutdown
- Exit with code 0 if both shut down cleanly
7. **Health monitoring:**
- Periodically check if processes are alive (poll returncode)
- If one process dies, log error and optionally shutdown other (per config flag)
- Display uptime and status every 30 seconds
8. **Logging and output:**
- Use Python logging module
- Log each process with [BACKEND] / [FRONTEND] prefix
- Merge stdout/stderr from both processes to terminal
- Show final status: "Backend running on http://localhost:{backend_port}, Frontend on http://localhost:{frontend_port}"
9. **Error handling:**
- If uvicorn not installed, error and suggest: `pip install uvicorn`
- If npm not found, error and suggest: install Node.js
- If ports already in use, error with port number
- If config files missing, log warnings but use defaults
10. **Required libraries:**
- sys, os, subprocess, signal, time (built-in)
- yaml (PyYAML)
- argparse (for CLI args --backend-only, --frontend-only)
- logging (for logging)
11. **Make executable:** `chmod +x scripts/run_standalone.py` with shebang: `#!/usr/bin/env python3`
</action>
<verify>
- `test -f scripts/run_standalone.py && head -1 scripts/run_standalone.py | grep -q "python3"` (shebang present)
- `test -x scripts/run_standalone.py` (executable)
- `python3 -m py_compile scripts/run_standalone.py` (valid Python syntax)
- `grep -q "import yaml" scripts/run_standalone.py` (PyYAML imported)
- `grep -q "config/backend.yaml\|config/frontend.yaml" scripts/run_standalone.py` (loads config)
- `grep -q "uvicorn\|subprocess.Popen" scripts/run_standalone.py` (backend startup present)
- `grep -q "npm\|node server" scripts/run_standalone.py` (frontend startup present)
- `grep -q "signal.signal\|SIGTERM" scripts/run_standalone.py` (signal handling present)
</verify>
<done>
scripts/run_standalone.py created with backend/frontend startup, YAML config parsing, process management, and graceful shutdown.
</done>
</task>
<task type="auto">
<name>Task 3: Create scripts/install_service.py (Systemd service installation)</name>
<files>
scripts/install_service.py
</files>
<read_first>
- install_service.sh (current bash implementation to port)
- inventory.service.template (systemd service template)
- config/backend.yaml.example (to understand config structure)
- config/network.yaml.example (for port information)
</read_first>
<action>
Create scripts/install_service.py to replace install_service.sh with Python implementation (per D-05).
**Key features:**
1. **Script signature:**
```bash
sudo python3 scripts/install_service.py [--user=service_user] [--port=port]
```
2. **Core functionality:**
- Read inventory.service.template (or create template inline)
- Load config from config/backend.yaml (for paths, ports)
- Generate systemd service file with correct paths and user/group
- Install service file to /etc/systemd/system/ainventory.service
- Enable service (systemctl enable)
- Display installation summary and next steps
3. **Config file parsing:**
- Load config/backend.yaml to get: data_dir, logs_dir
- Load config/network.yaml to get: backend_port (for documentation)
- Use defaults if missing
4. **Service file generation:**
- Read inventory.service.template
- Replace placeholders:
- {PROJECT_DIR}: current working directory (project root)
- {SERVICE_USER}: service user (default: www-data, configurable via --user)
- {BACKEND_PORT}: from config/network.yaml
- {DATA_DIR}: from config/backend.yaml
- {LOGS_DIR}: from config/backend.yaml
- Template should:
- Type=simple
- ExecStart=/usr/bin/python3 {PROJECT_DIR}/scripts/run_standalone.py --backend-only
- WorkingDirectory={PROJECT_DIR}
- User={SERVICE_USER}
- Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
- Restart=on-failure
- RestartSec=10
5. **Permission checks:**
- Require sudo/root: `if os.getuid() != 0: error("Must run with sudo")`
- Check project directory is readable
- Check data_dir and logs_dir exist or can be created
6. **Service file installation:**
- Write service file to /etc/systemd/system/ainventory.service
- Set permissions: 644 (readable, not writable by non-root)
- Run `systemctl daemon-reload`
- Run `systemctl enable ainventory.service`
- Optionally start service: `systemctl start ainventory.service`
7. **Status display:**
- Show service file location
- Show service user and group
- Show project directory
- Show next steps: `systemctl start ainventory`, `systemctl status ainventory`
- Show logs: `journalctl -u ainventory -f`
8. **Error handling:**
- Check if service already installed (offer --force to overwrite)
- Check if user exists (suggest: `useradd -r -s /bin/false {user}`)
- Check if directories are writable
- Descriptive errors with suggested fixes
9. **Required libraries:**
- sys, os, subprocess, pwd, grp (built-in)
- yaml (PyYAML)
- argparse (for CLI args)
- logging (for logging)
10. **Make executable:** `chmod +x scripts/install_service.py` with shebang: `#!/usr/bin/env python3`
</action>
<verify>
- `test -f scripts/install_service.py && head -1 scripts/install_service.py | grep -q "python3"` (shebang present)
- `test -x scripts/install_service.py` (executable)
- `python3 -m py_compile scripts/install_service.py` (valid Python syntax)
- `grep -q "import yaml" scripts/install_service.py` (PyYAML imported)
- `grep -q "config/backend.yaml\|config/network.yaml" scripts/install_service.py` (loads config)
- `grep -q "/etc/systemd/system\|systemctl" scripts/install_service.py` (systemd integration present)
- `grep -q "os.getuid\|sudo" scripts/install_service.py` (permission check present)
</verify>
<done>
scripts/install_service.py created with systemd service generation, config parsing, and installation logic.
</done>
</task>
<task type="auto">
<name>Task 4: Create scripts/export_prod.py (Production export/backup)</name>
<files>
scripts/export_prod.py
</files>
<read_first>
- export_prod.sh (current bash implementation to port)
- config/backend.yaml.example (for data_dir)
- DEPLOYMENT.md (backup procedures)
</read_first>
<action>
Create scripts/export_prod.py to replace export_prod.sh with Python implementation (per D-05).
**Key features:**
1. **Script signature:**
```bash
python3 scripts/export_prod.py [--output=/path/to/backup.tar.gz] [--include-logs]
```
2. **Core functionality:**
- Load config from config/backend.yaml (to find data_dir, logs_dir)
- Create tar.gz archive of production data
- Include database file(s), config files (no secrets), and optionally logs
- Output to specified location or default: backups/{timestamp}.tar.gz
- Display archive size and location
3. **Config file parsing:**
- Load config/backend.yaml to get: data_dir, logs_dir
- Use defaults if missing: data_dir=./data, logs_dir=./logs
4. **Archive creation:**
- Include: {data_dir}/* (all application data, database, etc.)
- Include: config/*.yaml.example (config templates)
- Include: config/backend.yaml, config/frontend.yaml, config/network.yaml (actual configs, no secrets)
- Include: config/secrets.yaml.example (secrets template only, NOT actual secrets.yaml)
- Include: logs/* (optional, if --include-logs flag)
- Exclude: config/secrets.yaml (never backup actual secrets)
- Exclude: node_modules/, __pycache__/, .git/, .venv/
- Exclude: temporary files, cache
5. **Archive naming:**
- Default: backups/ainventory_{timestamp}.tar.gz
- Timestamp format: YYYY-MM-DD_HH-MM-SS
- Custom path via --output flag
6. **Backup directory:**
- Create backups/ directory if not exists
- Set directory permissions: 750 (rwxr-x---)
7. **Verification:**
- Verify tar.gz was created successfully
- Display archive size: X.XX MB
- Display archive contents summary: "Includes database, data, and config (secrets excluded)"
8. **Error handling:**
- If data_dir doesn't exist, error and suggest creating it
- If no write permission to backups/, error and suggest location
- If tar command fails, show error and suggest troubleshooting
9. **Output example:**
```
[INFO] Loading config from config/backend.yaml
[INFO] Data directory: ./data
[INFO] Creating backup...
[INFO] Archive created: backups/ainventory_2026-04-23_14-30-45.tar.gz
[INFO] Archive size: 125.43 MB
[INFO] Contents: database, data, config (secrets excluded)
[INFO] Backup complete!
```
10. **Required libraries:**
- sys, os, subprocess, datetime, tarfile (built-in)
- yaml (PyYAML)
- argparse (for CLI args)
- logging (for logging)
11. **Make executable:** `chmod +x scripts/export_prod.py` with shebang: `#!/usr/bin/env python3`
</action>
<verify>
- `test -f scripts/export_prod.py && head -1 scripts/export_prod.py | grep -q "python3"` (shebang present)
- `test -x scripts/export_prod.py` (executable)
- `python3 -m py_compile scripts/export_prod.py` (valid Python syntax)
- `grep -q "import yaml" scripts/export_prod.py` (PyYAML imported)
- `grep -q "config/backend.yaml" scripts/export_prod.py` (loads config)
- `grep -q "tarfile\|tar.gz" scripts/export_prod.py` (tar archive creation present)
- `grep -q "secrets.yaml" scripts/export_prod.py | grep -q "exclude"` (secrets excluded from backup)
</verify>
<done>
scripts/export_prod.py created with production data export, YAML config parsing, archive creation, and secrets exclusion.
</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| User input → Script | Script arguments and config files must be validated |
| Script → System | Scripts execute system commands (subprocess) — must escape/quote properly |
| Script → Network | Health checks make HTTP requests (must handle timeouts) |
| Script → Filesystem | Scripts read/write files (must respect permissions) |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-07-09 | Injection | deploy.py subprocess | mitigate | Use subprocess with shell=False and list args (not f-strings). Example: `subprocess.run(["docker-compose", "up", "-d"], ...)` not `subprocess.run(f"docker-compose up -d", shell=True)`. |
| T-07-10 | Elevation of Privilege | install_service.py sudo | mitigate | Check `os.getuid() != 0` at start. Require sudo for systemd operations only. Log all systemctl calls. |
| T-07-11 | Information Disclosure | export_prod.py backup | mitigate | Exclude config/secrets.yaml explicitly in tarfile. Log what is excluded. Verify file permissions (backups/ dir 750). |
| T-07-12 | Denial of Service | Health checks timeout | mitigate | Set socket timeout to 10 seconds. Limit retry attempts to 12 (2 minutes total). Log timeout errors. |
</threat_model>
<verification>
**Phase 7, Plan 3 Verification Checklist:**
1. **scripts/deploy.py**
- [ ] File exists and is executable
- [ ] Shebang present: `#!/usr/bin/env python3`
- [ ] Loads config/docker.yaml and config/network.yaml
- [ ] Pre-flight checks for Docker, Docker Compose, config files
- [ ] Port availability checks implemented
- [ ] Health checks poll backend and frontend endpoints
- [ ] Color output for info/warning/success/error
- [ ] Displays deployment summary and access URLs
- [ ] Valid Python syntax
2. **scripts/run_standalone.py**
- [ ] File exists and is executable
- [ ] Shebang present: `#!/usr/bin/env python3`
- [ ] Loads config/backend.yaml and config/frontend.yaml
- [ ] Launches uvicorn for backend with correct port and settings
- [ ] Launches frontend (npm dev or node server.js) with correct port
- [ ] Signal handling (SIGTERM/SIGINT) for clean shutdown
- [ ] Process monitoring and output display with prefixes
- [ ] Valid Python syntax
3. **scripts/install_service.py**
- [ ] File exists and is executable
- [ ] Shebang present: `#!/usr/bin/env python3`
- [ ] Checks for sudo/root permission
- [ ] Loads config/backend.yaml and config/network.yaml
- [ ] Generates systemd service file from template
- [ ] Replaces placeholders: {PROJECT_DIR}, {SERVICE_USER}, {BACKEND_PORT}, etc.
- [ ] Installs to /etc/systemd/system/ with correct permissions
- [ ] Runs systemctl daemon-reload and enable
- [ ] Valid Python syntax
4. **scripts/export_prod.py**
- [ ] File exists and is executable
- [ ] Shebang present: `#!/usr/bin/env python3`
- [ ] Loads config/backend.yaml to find data_dir
- [ ] Creates tar.gz archive with data and config files
- [ ] Excludes config/secrets.yaml (actual secrets, not example)
- [ ] Includes config/*.yaml.example files
- [ ] Output to backups/{timestamp}.tar.gz or custom path
- [ ] Displays archive size and summary
- [ ] Valid Python syntax
5. **Security (subprocess, permissions, file ops)**
- [ ] All subprocess calls use shell=False with list args
- [ ] No f-strings in shell commands
- [ ] File operations respect umask/permissions
- [ ] No hardcoded credentials in scripts
6. **Integration**
- [ ] Each script loads YAML config files correctly
- [ ] Scripts reference new config/ structure (not inventory.env)
- [ ] Error messages are helpful and actionable
</verification>
<success_criteria>
- 4 Python scripts created (deploy.py, run_standalone.py, install_service.py, export_prod.py)
- All scripts use PyYAML to parse config files
- All scripts are executable with proper shebangs
- deploy.py handles Docker deployment with health checks
- run_standalone.py launches backend and frontend without Docker
- install_service.py creates systemd service with new config paths
- export_prod.py exports production data excluding secrets
- All subprocess calls use shell=False (secure)
- Error handling and logging present in all scripts
- Scripts integrate with new config/ structure (D-05, D-06, D-07)
</success_criteria>
<output>
After completion, create `.planning/phases/07-config-consolidation/07-03-SUMMARY.md`
</output>
Reference in New Issue View Git Blame Copy Permalink