feat(6): phase 6 plan 02 - operational runbook and documentation
- Created OPERATIONAL_RUNBOOK.md: comprehensive step-by-step procedures for both Docker and Standalone deployment modes covering deployment, daily ops, troubleshooting, backup/restore, disaster recovery, scaling, and updates - Created HEALTH_MONITORING_CHECKLIST.md: daily/weekly/monthly health check procedures with alert thresholds and quick troubleshooting reference - Created DISASTER_RECOVERY_PLAN.md: detailed procedures for 6 failure scenarios (database corruption, hardware failure, data center failure, app crash, disk full, network isolation) with RTO/RPO targets - Created CONFIGURATION_REFERENCE.md: complete documentation of all inventory.env parameters for both deployment modes with common scenarios and troubleshooting - Created EMERGENCY_PROCEDURES.md: quick-reference incident response playbook with 7 critical scenarios, decision tree, escalation path, and printable cheat sheet - Created scripts/backup.sh: automated backup script supporting both Docker and Standalone with integrity verification and retention management - Created scripts/restore.sh: restore script with triple confirmation, safety backups, and validation tests for both deployment modes - Created config/backup-cron.sh: installer for daily/weekly automated backup cron jobs (2 AM daily, 3 AM Sunday) All documentation covers dual-deployment modes with shared configuration files. Documentation is operator-ready with copy-paste commands and clear expected outputs.
This commit is contained in:
480
docs/OPERATIONAL_RUNBOOK.md
Normal file
480
docs/OPERATIONAL_RUNBOOK.md
Normal file
@@ -0,0 +1,480 @@
|
||||
# Operational Runbook
|
||||
|
||||
**Audience**: Systems operators, site managers, DevOps teams
|
||||
**Target**: Minimal training required; step-by-step procedures
|
||||
**Last Updated**: 2026-04-22
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This runbook covers operational procedures for both Docker and Standalone deployment modes. Both modes use the same configuration files (inventory.env) and backup/restore scripts.
|
||||
|
||||
---
|
||||
|
||||
## 1. Initial Deployment
|
||||
|
||||
### Requirements
|
||||
|
||||
- Ubuntu 22.04 LTS or similar
|
||||
- For Docker mode: Docker and Docker Compose installed
|
||||
- For Standalone mode: Python 3.12+, Node.js 18+, npm
|
||||
- 2GB RAM minimum, 10GB disk (recommended: 4GB/50GB for production)
|
||||
- Internet access (first-time setup only)
|
||||
|
||||
### Docker Deployment Steps
|
||||
|
||||
1. **Clone repository**
|
||||
```bash
|
||||
git clone <repo_url> /opt/tfm-inventory
|
||||
cd /opt/tfm-inventory
|
||||
```
|
||||
|
||||
2. **Configure environment**
|
||||
```bash
|
||||
cp inventory.env.template inventory.env
|
||||
# Edit inventory.env with your settings:
|
||||
# - BACKEND_PORT=8000, FRONTEND_PORT=3000
|
||||
# - JWT_SECRET_KEY (generate: openssl rand -hex 32)
|
||||
# - AI settings (Gemini/Claude API keys, optional)
|
||||
# - LDAP settings (if using enterprise auth)
|
||||
```
|
||||
|
||||
3. **Deploy**
|
||||
```bash
|
||||
chmod +x deploy.sh scripts/backup.sh scripts/restore.sh
|
||||
./deploy.sh production
|
||||
```
|
||||
|
||||
4. **Verify deployment**
|
||||
- Frontend: http://your-server:3000
|
||||
- Backend API: http://your-server:8000
|
||||
- API Docs: http://your-server:8000/docs
|
||||
- Health check: `curl http://localhost:8000/health`
|
||||
|
||||
5. **Create admin user** (if not auto-created)
|
||||
```bash
|
||||
docker-compose exec backend python -c "
|
||||
from backend.db import SessionLocal, User
|
||||
db = SessionLocal()
|
||||
user = User(username='admin', hashed_password='...', is_admin=True)
|
||||
db.add(user)
|
||||
db.commit()
|
||||
"
|
||||
```
|
||||
|
||||
### Standalone Deployment Steps
|
||||
|
||||
1. **Clone repository**
|
||||
```bash
|
||||
git clone <repo_url> /opt/tfm-inventory
|
||||
cd /opt/tfm-inventory
|
||||
```
|
||||
|
||||
2. **Configure environment**
|
||||
```bash
|
||||
cp inventory.env.template inventory.env
|
||||
# Edit with same settings as Docker mode
|
||||
```
|
||||
|
||||
3. **Install dependencies**
|
||||
```bash
|
||||
# Backend
|
||||
cd backend
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
cd ..
|
||||
|
||||
# Frontend
|
||||
cd frontend
|
||||
npm install
|
||||
cd ..
|
||||
```
|
||||
|
||||
4. **Deploy**
|
||||
```bash
|
||||
chmod +x start_server.sh scripts/backup.sh scripts/restore.sh
|
||||
./start_server.sh
|
||||
```
|
||||
|
||||
5. **Verify deployment**
|
||||
- Frontend: http://localhost:3000
|
||||
- Backend API: http://localhost:8000
|
||||
- Health check: `curl http://localhost:8000/health`
|
||||
|
||||
---
|
||||
|
||||
## 2. Daily Operations
|
||||
|
||||
### Health Checks (Daily, ~5 minutes)
|
||||
|
||||
**Docker mode:**
|
||||
```bash
|
||||
# Check all services
|
||||
docker-compose ps
|
||||
# Expected: All services "Up"
|
||||
|
||||
# Check API health
|
||||
curl http://localhost:8000/health
|
||||
# Expected: 200 OK, response time <100ms
|
||||
|
||||
# Check database
|
||||
du -h data/inventory.db
|
||||
|
||||
# Check for errors
|
||||
docker-compose logs | grep ERROR | tail -5
|
||||
```
|
||||
|
||||
**Standalone mode:**
|
||||
```bash
|
||||
# Check processes
|
||||
ps aux | grep -E "(uvicorn|next)" | grep -v grep
|
||||
|
||||
# Check API health
|
||||
curl http://localhost:8000/health
|
||||
|
||||
# Check database
|
||||
du -h data/inventory.db
|
||||
|
||||
# Check logs
|
||||
tail -50 logs/backend.log logs/frontend.log 2>/dev/null
|
||||
```
|
||||
|
||||
### Backup (Automated)
|
||||
|
||||
```bash
|
||||
# Verify automatic backup ran (cron jobs)
|
||||
ls -lh backups/ | head -1
|
||||
# Expected: File timestamp within last 24 hours
|
||||
|
||||
# Manual backup (if needed)
|
||||
./scripts/backup.sh manual
|
||||
|
||||
# View backup schedule
|
||||
sudo crontab -l | grep backup
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
**Docker mode:**
|
||||
```bash
|
||||
# Real-time logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Backend performance
|
||||
docker stats --no-stream | grep backend
|
||||
|
||||
# Database status
|
||||
docker-compose exec backend sqlite3 /app/data/inventory.db \
|
||||
"SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"
|
||||
```
|
||||
|
||||
**Standalone mode:**
|
||||
```bash
|
||||
# Real-time logs
|
||||
tail -f logs/backend.log logs/frontend.log
|
||||
|
||||
# System resources
|
||||
top -p $(pgrep -f uvicorn | head -1)
|
||||
|
||||
# Database status
|
||||
sqlite3 data/inventory.db \
|
||||
"SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Troubleshooting
|
||||
|
||||
### Service Won't Start
|
||||
|
||||
**Docker mode:**
|
||||
```bash
|
||||
# Check Docker daemon
|
||||
docker ps
|
||||
|
||||
# Check port conflicts
|
||||
netstat -tuln | grep -E "8000|3000|8906|8907"
|
||||
|
||||
# View service logs
|
||||
docker-compose logs backend
|
||||
docker-compose logs frontend
|
||||
docker-compose logs proxy
|
||||
```
|
||||
|
||||
**Standalone mode:**
|
||||
```bash
|
||||
# Check if processes are running
|
||||
ps aux | grep -E "(uvicorn|next)"
|
||||
|
||||
# Check port conflicts
|
||||
netstat -tuln | grep -E "8000|3000"
|
||||
|
||||
# Check logs
|
||||
cat logs/backend.log | tail -50
|
||||
```
|
||||
|
||||
### High CPU/Memory
|
||||
|
||||
**Docker mode:**
|
||||
```bash
|
||||
# Identify container
|
||||
docker stats --no-stream
|
||||
|
||||
# Restart container
|
||||
docker-compose restart backend
|
||||
|
||||
# Check for slow queries
|
||||
docker-compose logs backend | grep "slow"
|
||||
```
|
||||
|
||||
**Standalone mode:**
|
||||
```bash
|
||||
# Kill and restart
|
||||
pkill -f uvicorn
|
||||
pkill -f "next start"
|
||||
sleep 2
|
||||
./start_server.sh
|
||||
```
|
||||
|
||||
### Database Locked
|
||||
|
||||
**Docker mode:**
|
||||
```bash
|
||||
docker-compose restart backend
|
||||
# Wait 30 seconds
|
||||
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA journal_mode;"
|
||||
```
|
||||
|
||||
**Standalone mode:**
|
||||
```bash
|
||||
pkill -f uvicorn
|
||||
sleep 2
|
||||
# Restart backend only (no need for frontend restart)
|
||||
cd backend && source venv/bin/activate && uvicorn main:app --host 0.0.0.0 --port 8000 &
|
||||
```
|
||||
|
||||
### HTTPS Certificate Issues
|
||||
|
||||
**Docker mode:**
|
||||
```bash
|
||||
# Certificates regenerated automatically
|
||||
# If issues persist:
|
||||
rm -rf data/caddy_*
|
||||
docker-compose restart proxy
|
||||
# Wait 30 seconds for new certs to generate
|
||||
```
|
||||
|
||||
**Standalone mode:**
|
||||
```bash
|
||||
# For local development/testing, HTTP is sufficient
|
||||
# For production HTTPS, configure reverse proxy (nginx/Caddy) separately
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Backup & Restore
|
||||
|
||||
### Automated Backups
|
||||
|
||||
```bash
|
||||
# Verify cron jobs are installed
|
||||
sudo bash config/backup-cron.sh
|
||||
|
||||
# View backup history
|
||||
ls -lh backups/
|
||||
|
||||
# Check backup log
|
||||
tail -20 logs/backup-daily.log
|
||||
```
|
||||
|
||||
**Backup Schedule:**
|
||||
- Daily: 2 AM, retention 30 days
|
||||
- Weekly: 3 AM Sundays, retention 90 days
|
||||
|
||||
### Manual Backup
|
||||
|
||||
```bash
|
||||
# Create backup
|
||||
./scripts/backup.sh manual
|
||||
|
||||
# Verify backup created and is valid
|
||||
tar -tzf backups/inventory-*.tar.gz | head
|
||||
```
|
||||
|
||||
### Manual Restore
|
||||
|
||||
```bash
|
||||
# List available backups
|
||||
ls backups/
|
||||
|
||||
# Restore specific backup (Docker mode)
|
||||
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz --validate
|
||||
|
||||
# Restore specific backup (Standalone mode)
|
||||
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz
|
||||
# Then restart: ./start_server.sh
|
||||
|
||||
# Validate data after restore
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
**Recovery Objectives:**
|
||||
- RTO (Recovery Time): <10 minutes
|
||||
- RPO (Recovery Point): 1 day (daily backup)
|
||||
|
||||
---
|
||||
|
||||
## 5. Disaster Recovery
|
||||
|
||||
### Complete System Failure (Hardware or Data Corruption)
|
||||
|
||||
**For Docker:**
|
||||
1. Provision new Ubuntu 22.04 LTS server (same specs)
|
||||
2. Clone repository: `git clone <repo> /opt/tfm-inventory && cd /opt/tfm-inventory`
|
||||
3. Copy latest backup from offsite or previous backup directory
|
||||
4. Restore: `./scripts/restore.sh /path/to/backup.tar.gz --validate`
|
||||
5. Update DNS/load balancer to new server IP
|
||||
6. Verify all services healthy and data present
|
||||
|
||||
**For Standalone:**
|
||||
1. Provision new Ubuntu 22.04 LTS server
|
||||
2. Clone repository
|
||||
3. Install dependencies (Python venv, Node.js)
|
||||
4. Copy latest backup
|
||||
5. Restore: `./scripts/restore.sh /path/to/backup.tar.gz`
|
||||
6. Start services: `./start_server.sh`
|
||||
7. Verify connectivity and data
|
||||
|
||||
**RTO**: <30 minutes (provisioning + restore)
|
||||
**RPO**: 1 day (latest backup)
|
||||
|
||||
### Data Center Failure
|
||||
|
||||
1. Activate secondary site or failover to cloud
|
||||
2. Clone repository on new infrastructure
|
||||
3. Restore latest backup: `./scripts/restore.sh backup.tar.gz --validate`
|
||||
4. Update DNS to new location
|
||||
5. Notify users of recovery (1-day data loss acceptable)
|
||||
|
||||
---
|
||||
|
||||
## 6. Scaling Operations
|
||||
|
||||
### Adding Users (5+ concurrent)
|
||||
|
||||
Current configuration supports 5 concurrent users safely.
|
||||
|
||||
**Docker mode:**
|
||||
```bash
|
||||
# Increase backend memory
|
||||
# Edit docker-compose.yml:
|
||||
# backend:
|
||||
# mem_limit: 4g
|
||||
|
||||
# Increase database connections
|
||||
docker-compose exec backend \
|
||||
python -c "import backend.config; print(backend.config.DB_POOL_SIZE)"
|
||||
```
|
||||
|
||||
**Standalone mode:**
|
||||
```bash
|
||||
# Increase Python process resources
|
||||
# Edit start_server.sh to add workers/processes if using Gunicorn
|
||||
|
||||
# Monitor memory usage
|
||||
ps aux | grep uvicorn
|
||||
```
|
||||
|
||||
### Database Growth (10K+ items)
|
||||
|
||||
As inventory grows beyond 10K items:
|
||||
1. Monitor query performance: `PRAGMA optimize;`
|
||||
2. Create indexes on frequently searched columns
|
||||
3. Vacuum database: `VACUUM;`
|
||||
4. Consider archiving old audit logs (v3 feature)
|
||||
|
||||
---
|
||||
|
||||
## 7. Updates & Upgrades
|
||||
|
||||
### Patch Update (v1.14.x → v1.14.y)
|
||||
|
||||
```bash
|
||||
# Backup first
|
||||
./scripts/backup.sh manual
|
||||
|
||||
# Pull latest code
|
||||
git pull origin main
|
||||
|
||||
# Docker mode:
|
||||
./deploy.sh production --rebuild
|
||||
|
||||
# Standalone mode:
|
||||
pkill -f uvicorn
|
||||
pkill -f "next start"
|
||||
cd backend && source venv/bin/activate && pip install -r requirements.txt
|
||||
cd ../frontend && npm install && npm run build
|
||||
./start_server.sh
|
||||
|
||||
# Verify
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
### Major Update (v1.x → v2.x)
|
||||
|
||||
```bash
|
||||
# Create backup before proceeding
|
||||
./scripts/backup.sh manual
|
||||
|
||||
# Review CHANGELOG for breaking changes
|
||||
cat CHANGELOG.md | grep "v2.0"
|
||||
|
||||
# Test in staging first (restore backup there)
|
||||
./scripts/restore.sh backups/production.tar.gz
|
||||
|
||||
# If staging successful, proceed to production
|
||||
git checkout v2.0
|
||||
./deploy.sh production --rebuild # Docker mode
|
||||
# OR
|
||||
./start_server.sh # Standalone mode
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Performance Baseline
|
||||
|
||||
- Backend: <100ms API response time at 5 concurrent users
|
||||
- Frontend: <1s page load
|
||||
- Database: <500 queries/min with 10K items
|
||||
- Memory: Backend <500MB, Frontend <200MB
|
||||
- CPU: Both services <70% usage under normal load
|
||||
|
||||
---
|
||||
|
||||
## 9. Emergency Contacts & Escalation
|
||||
|
||||
- **Developer Support**: dev@example.com
|
||||
- **Infrastructure**: ops@example.com
|
||||
- **24/7 On-call**: [contact info]
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Quick Reference
|
||||
|
||||
| Task | Docker Command | Standalone Command |
|
||||
|------|---|---|
|
||||
| Health Check | `docker-compose ps` | `ps aux \| grep -E "(uvicorn\|next)"` |
|
||||
| View Logs | `docker-compose logs -f` | `tail -f logs/*.log` |
|
||||
| Restart Backend | `docker-compose restart backend` | `pkill -f uvicorn; ./start_server.sh` |
|
||||
| Backup | `./scripts/backup.sh manual` | `./scripts/backup.sh manual` |
|
||||
| Restore | `./scripts/restore.sh file.tar.gz` | `./scripts/restore.sh file.tar.gz` |
|
||||
| Stop Services | `docker-compose down` | `pkill -f uvicorn; pkill -f "next"` |
|
||||
|
||||
---
|
||||
|
||||
**Version**: 1.0
|
||||
**Last Updated**: 2026-04-22
|
||||
**Maintained By**: Operations Team
|
||||
**Next Review**: 2026-05-22
|
||||
Reference in New Issue
Block a user