Daniel Bedeleanu
fc149184e9
- Created OPERATIONAL_RUNBOOK.md: comprehensive step-by-step procedures for both Docker and Standalone deployment modes covering deployment, daily ops, troubleshooting, backup/restore, disaster recovery, scaling, and updates - Created HEALTH_MONITORING_CHECKLIST.md: daily/weekly/monthly health check procedures with alert thresholds and quick troubleshooting reference - Created DISASTER_RECOVERY_PLAN.md: detailed procedures for 6 failure scenarios (database corruption, hardware failure, data center failure, app crash, disk full, network isolation) with RTO/RPO targets - Created CONFIGURATION_REFERENCE.md: complete documentation of all inventory.env parameters for both deployment modes with common scenarios and troubleshooting - Created EMERGENCY_PROCEDURES.md: quick-reference incident response playbook with 7 critical scenarios, decision tree, escalation path, and printable cheat sheet - Created scripts/backup.sh: automated backup script supporting both Docker and Standalone with integrity verification and retention management - Created scripts/restore.sh: restore script with triple confirmation, safety backups, and validation tests for both deployment modes - Created config/backup-cron.sh: installer for daily/weekly automated backup cron jobs (2 AM daily, 3 AM Sunday) All documentation covers dual-deployment modes with shared configuration files. Documentation is operator-ready with copy-paste commands and clear expected outputs.
481 lines
10 KiB
Markdown
481 lines
10 KiB
Markdown
# Operational Runbook
|
|
|
|
**Audience**: Systems operators, site managers, DevOps teams
|
|
**Target**: Minimal training required; step-by-step procedures
|
|
**Last Updated**: 2026-04-22
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
This runbook covers operational procedures for both Docker and Standalone deployment modes. Both modes use the same configuration files (inventory.env) and backup/restore scripts.
|
|
|
|
---
|
|
|
|
## 1. Initial Deployment
|
|
|
|
### Requirements
|
|
|
|
- Ubuntu 22.04 LTS or similar
|
|
- For Docker mode: Docker and Docker Compose installed
|
|
- For Standalone mode: Python 3.12+, Node.js 18+, npm
|
|
- 2GB RAM minimum, 10GB disk (recommended: 4GB/50GB for production)
|
|
- Internet access (first-time setup only)
|
|
|
|
### Docker Deployment Steps
|
|
|
|
1. **Clone repository**
|
|
```bash
|
|
git clone <repo_url> /opt/tfm-inventory
|
|
cd /opt/tfm-inventory
|
|
```
|
|
|
|
2. **Configure environment**
|
|
```bash
|
|
cp inventory.env.template inventory.env
|
|
# Edit inventory.env with your settings:
|
|
# - BACKEND_PORT=8000, FRONTEND_PORT=3000
|
|
# - JWT_SECRET_KEY (generate: openssl rand -hex 32)
|
|
# - AI settings (Gemini/Claude API keys, optional)
|
|
# - LDAP settings (if using enterprise auth)
|
|
```
|
|
|
|
3. **Deploy**
|
|
```bash
|
|
chmod +x deploy.sh scripts/backup.sh scripts/restore.sh
|
|
./deploy.sh production
|
|
```
|
|
|
|
4. **Verify deployment**
|
|
- Frontend: http://your-server:3000
|
|
- Backend API: http://your-server:8000
|
|
- API Docs: http://your-server:8000/docs
|
|
- Health check: `curl http://localhost:8000/health`
|
|
|
|
5. **Create admin user** (if not auto-created)
|
|
```bash
|
|
docker-compose exec backend python -c "
|
|
from backend.db import SessionLocal, User
|
|
db = SessionLocal()
|
|
user = User(username='admin', hashed_password='...', is_admin=True)
|
|
db.add(user)
|
|
db.commit()
|
|
"
|
|
```
|
|
|
|
### Standalone Deployment Steps
|
|
|
|
1. **Clone repository**
|
|
```bash
|
|
git clone <repo_url> /opt/tfm-inventory
|
|
cd /opt/tfm-inventory
|
|
```
|
|
|
|
2. **Configure environment**
|
|
```bash
|
|
cp inventory.env.template inventory.env
|
|
# Edit with same settings as Docker mode
|
|
```
|
|
|
|
3. **Install dependencies**
|
|
```bash
|
|
# Backend
|
|
cd backend
|
|
python3 -m venv venv
|
|
source venv/bin/activate
|
|
pip install -r requirements.txt
|
|
cd ..
|
|
|
|
# Frontend
|
|
cd frontend
|
|
npm install
|
|
cd ..
|
|
```
|
|
|
|
4. **Deploy**
|
|
```bash
|
|
chmod +x start_server.sh scripts/backup.sh scripts/restore.sh
|
|
./start_server.sh
|
|
```
|
|
|
|
5. **Verify deployment**
|
|
- Frontend: http://localhost:3000
|
|
- Backend API: http://localhost:8000
|
|
- Health check: `curl http://localhost:8000/health`
|
|
|
|
---
|
|
|
|
## 2. Daily Operations
|
|
|
|
### Health Checks (Daily, ~5 minutes)
|
|
|
|
**Docker mode:**
|
|
```bash
|
|
# Check all services
|
|
docker-compose ps
|
|
# Expected: All services "Up"
|
|
|
|
# Check API health
|
|
curl http://localhost:8000/health
|
|
# Expected: 200 OK, response time <100ms
|
|
|
|
# Check database
|
|
du -h data/inventory.db
|
|
|
|
# Check for errors
|
|
docker-compose logs | grep ERROR | tail -5
|
|
```
|
|
|
|
**Standalone mode:**
|
|
```bash
|
|
# Check processes
|
|
ps aux | grep -E "(uvicorn|next)" | grep -v grep
|
|
|
|
# Check API health
|
|
curl http://localhost:8000/health
|
|
|
|
# Check database
|
|
du -h data/inventory.db
|
|
|
|
# Check logs
|
|
tail -50 logs/backend.log logs/frontend.log 2>/dev/null
|
|
```
|
|
|
|
### Backup (Automated)
|
|
|
|
```bash
|
|
# Verify automatic backup ran (cron jobs)
|
|
ls -lh backups/ | head -1
|
|
# Expected: File timestamp within last 24 hours
|
|
|
|
# Manual backup (if needed)
|
|
./scripts/backup.sh manual
|
|
|
|
# View backup schedule
|
|
sudo crontab -l | grep backup
|
|
```
|
|
|
|
### Monitoring
|
|
|
|
**Docker mode:**
|
|
```bash
|
|
# Real-time logs
|
|
docker-compose logs -f
|
|
|
|
# Backend performance
|
|
docker stats --no-stream | grep backend
|
|
|
|
# Database status
|
|
docker-compose exec backend sqlite3 /app/data/inventory.db \
|
|
"SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"
|
|
```
|
|
|
|
**Standalone mode:**
|
|
```bash
|
|
# Real-time logs
|
|
tail -f logs/backend.log logs/frontend.log
|
|
|
|
# System resources
|
|
top -p $(pgrep -f uvicorn | head -1)
|
|
|
|
# Database status
|
|
sqlite3 data/inventory.db \
|
|
"SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Troubleshooting
|
|
|
|
### Service Won't Start
|
|
|
|
**Docker mode:**
|
|
```bash
|
|
# Check Docker daemon
|
|
docker ps
|
|
|
|
# Check port conflicts
|
|
netstat -tuln | grep -E "8000|3000|8906|8907"
|
|
|
|
# View service logs
|
|
docker-compose logs backend
|
|
docker-compose logs frontend
|
|
docker-compose logs proxy
|
|
```
|
|
|
|
**Standalone mode:**
|
|
```bash
|
|
# Check if processes are running
|
|
ps aux | grep -E "(uvicorn|next)"
|
|
|
|
# Check port conflicts
|
|
netstat -tuln | grep -E "8000|3000"
|
|
|
|
# Check logs
|
|
cat logs/backend.log | tail -50
|
|
```
|
|
|
|
### High CPU/Memory
|
|
|
|
**Docker mode:**
|
|
```bash
|
|
# Identify container
|
|
docker stats --no-stream
|
|
|
|
# Restart container
|
|
docker-compose restart backend
|
|
|
|
# Check for slow queries
|
|
docker-compose logs backend | grep "slow"
|
|
```
|
|
|
|
**Standalone mode:**
|
|
```bash
|
|
# Kill and restart
|
|
pkill -f uvicorn
|
|
pkill -f "next start"
|
|
sleep 2
|
|
./start_server.sh
|
|
```
|
|
|
|
### Database Locked
|
|
|
|
**Docker mode:**
|
|
```bash
|
|
docker-compose restart backend
|
|
# Wait 30 seconds
|
|
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA journal_mode;"
|
|
```
|
|
|
|
**Standalone mode:**
|
|
```bash
|
|
pkill -f uvicorn
|
|
sleep 2
|
|
# Restart backend only (no need for frontend restart)
|
|
cd backend && source venv/bin/activate && uvicorn main:app --host 0.0.0.0 --port 8000 &
|
|
```
|
|
|
|
### HTTPS Certificate Issues
|
|
|
|
**Docker mode:**
|
|
```bash
|
|
# Certificates regenerated automatically
|
|
# If issues persist:
|
|
rm -rf data/caddy_*
|
|
docker-compose restart proxy
|
|
# Wait 30 seconds for new certs to generate
|
|
```
|
|
|
|
**Standalone mode:**
|
|
```bash
|
|
# For local development/testing, HTTP is sufficient
|
|
# For production HTTPS, configure reverse proxy (nginx/Caddy) separately
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Backup & Restore
|
|
|
|
### Automated Backups
|
|
|
|
```bash
|
|
# Verify cron jobs are installed
|
|
sudo bash config/backup-cron.sh
|
|
|
|
# View backup history
|
|
ls -lh backups/
|
|
|
|
# Check backup log
|
|
tail -20 logs/backup-daily.log
|
|
```
|
|
|
|
**Backup Schedule:**
|
|
- Daily: 2 AM, retention 30 days
|
|
- Weekly: 3 AM Sundays, retention 90 days
|
|
|
|
### Manual Backup
|
|
|
|
```bash
|
|
# Create backup
|
|
./scripts/backup.sh manual
|
|
|
|
# Verify backup created and is valid
|
|
tar -tzf backups/inventory-*.tar.gz | head
|
|
```
|
|
|
|
### Manual Restore
|
|
|
|
```bash
|
|
# List available backups
|
|
ls backups/
|
|
|
|
# Restore specific backup (Docker mode)
|
|
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz --validate
|
|
|
|
# Restore specific backup (Standalone mode)
|
|
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz
|
|
# Then restart: ./start_server.sh
|
|
|
|
# Validate data after restore
|
|
curl http://localhost:8000/health
|
|
```
|
|
|
|
**Recovery Objectives:**
|
|
- RTO (Recovery Time): <10 minutes
|
|
- RPO (Recovery Point): 1 day (daily backup)
|
|
|
|
---
|
|
|
|
## 5. Disaster Recovery
|
|
|
|
### Complete System Failure (Hardware or Data Corruption)
|
|
|
|
**For Docker:**
|
|
1. Provision new Ubuntu 22.04 LTS server (same specs)
|
|
2. Clone repository: `git clone <repo> /opt/tfm-inventory && cd /opt/tfm-inventory`
|
|
3. Copy latest backup from offsite or previous backup directory
|
|
4. Restore: `./scripts/restore.sh /path/to/backup.tar.gz --validate`
|
|
5. Update DNS/load balancer to new server IP
|
|
6. Verify all services healthy and data present
|
|
|
|
**For Standalone:**
|
|
1. Provision new Ubuntu 22.04 LTS server
|
|
2. Clone repository
|
|
3. Install dependencies (Python venv, Node.js)
|
|
4. Copy latest backup
|
|
5. Restore: `./scripts/restore.sh /path/to/backup.tar.gz`
|
|
6. Start services: `./start_server.sh`
|
|
7. Verify connectivity and data
|
|
|
|
**RTO**: <30 minutes (provisioning + restore)
|
|
**RPO**: 1 day (latest backup)
|
|
|
|
### Data Center Failure
|
|
|
|
1. Activate secondary site or failover to cloud
|
|
2. Clone repository on new infrastructure
|
|
3. Restore latest backup: `./scripts/restore.sh backup.tar.gz --validate`
|
|
4. Update DNS to new location
|
|
5. Notify users of recovery (1-day data loss acceptable)
|
|
|
|
---
|
|
|
|
## 6. Scaling Operations
|
|
|
|
### Adding Users (5+ concurrent)
|
|
|
|
Current configuration supports 5 concurrent users safely.
|
|
|
|
**Docker mode:**
|
|
```bash
|
|
# Increase backend memory
|
|
# Edit docker-compose.yml:
|
|
# backend:
|
|
# mem_limit: 4g
|
|
|
|
# Increase database connections
|
|
docker-compose exec backend \
|
|
python -c "import backend.config; print(backend.config.DB_POOL_SIZE)"
|
|
```
|
|
|
|
**Standalone mode:**
|
|
```bash
|
|
# Increase Python process resources
|
|
# Edit start_server.sh to add workers/processes if using Gunicorn
|
|
|
|
# Monitor memory usage
|
|
ps aux | grep uvicorn
|
|
```
|
|
|
|
### Database Growth (10K+ items)
|
|
|
|
As inventory grows beyond 10K items:
|
|
1. Monitor query performance: `PRAGMA optimize;`
|
|
2. Create indexes on frequently searched columns
|
|
3. Vacuum database: `VACUUM;`
|
|
4. Consider archiving old audit logs (v3 feature)
|
|
|
|
---
|
|
|
|
## 7. Updates & Upgrades
|
|
|
|
### Patch Update (v1.14.x → v1.14.y)
|
|
|
|
```bash
|
|
# Backup first
|
|
./scripts/backup.sh manual
|
|
|
|
# Pull latest code
|
|
git pull origin main
|
|
|
|
# Docker mode:
|
|
./deploy.sh production --rebuild
|
|
|
|
# Standalone mode:
|
|
pkill -f uvicorn
|
|
pkill -f "next start"
|
|
cd backend && source venv/bin/activate && pip install -r requirements.txt
|
|
cd ../frontend && npm install && npm run build
|
|
./start_server.sh
|
|
|
|
# Verify
|
|
curl http://localhost:8000/health
|
|
```
|
|
|
|
### Major Update (v1.x → v2.x)
|
|
|
|
```bash
|
|
# Create backup before proceeding
|
|
./scripts/backup.sh manual
|
|
|
|
# Review CHANGELOG for breaking changes
|
|
cat CHANGELOG.md | grep "v2.0"
|
|
|
|
# Test in staging first (restore backup there)
|
|
./scripts/restore.sh backups/production.tar.gz
|
|
|
|
# If staging successful, proceed to production
|
|
git checkout v2.0
|
|
./deploy.sh production --rebuild # Docker mode
|
|
# OR
|
|
./start_server.sh # Standalone mode
|
|
```
|
|
|
|
---
|
|
|
|
## 8. Performance Baseline
|
|
|
|
- Backend: <100ms API response time at 5 concurrent users
|
|
- Frontend: <1s page load
|
|
- Database: <500 queries/min with 10K items
|
|
- Memory: Backend <500MB, Frontend <200MB
|
|
- CPU: Both services <70% usage under normal load
|
|
|
|
---
|
|
|
|
## 9. Emergency Contacts & Escalation
|
|
|
|
- **Developer Support**: dev@example.com
|
|
- **Infrastructure**: ops@example.com
|
|
- **24/7 On-call**: [contact info]
|
|
|
|
---
|
|
|
|
## Appendix: Quick Reference
|
|
|
|
| Task | Docker Command | Standalone Command |
|
|
|------|---|---|
|
|
| Health Check | `docker-compose ps` | `ps aux \| grep -E "(uvicorn\|next)"` |
|
|
| View Logs | `docker-compose logs -f` | `tail -f logs/*.log` |
|
|
| Restart Backend | `docker-compose restart backend` | `pkill -f uvicorn; ./start_server.sh` |
|
|
| Backup | `./scripts/backup.sh manual` | `./scripts/backup.sh manual` |
|
|
| Restore | `./scripts/restore.sh file.tar.gz` | `./scripts/restore.sh file.tar.gz` |
|
|
| Stop Services | `docker-compose down` | `pkill -f uvicorn; pkill -f "next"` |
|
|
|
|
---
|
|
|
|
**Version**: 1.0
|
|
**Last Updated**: 2026-04-22
|
|
**Maintained By**: Operations Team
|
|
**Next Review**: 2026-05-22
|