feat(6): phase 6 plan 02 - operational runbook and documentation
- Created OPERATIONAL_RUNBOOK.md: comprehensive step-by-step procedures for both Docker and Standalone deployment modes covering deployment, daily ops, troubleshooting, backup/restore, disaster recovery, scaling, and updates - Created HEALTH_MONITORING_CHECKLIST.md: daily/weekly/monthly health check procedures with alert thresholds and quick troubleshooting reference - Created DISASTER_RECOVERY_PLAN.md: detailed procedures for 6 failure scenarios (database corruption, hardware failure, data center failure, app crash, disk full, network isolation) with RTO/RPO targets - Created CONFIGURATION_REFERENCE.md: complete documentation of all inventory.env parameters for both deployment modes with common scenarios and troubleshooting - Created EMERGENCY_PROCEDURES.md: quick-reference incident response playbook with 7 critical scenarios, decision tree, escalation path, and printable cheat sheet - Created scripts/backup.sh: automated backup script supporting both Docker and Standalone with integrity verification and retention management - Created scripts/restore.sh: restore script with triple confirmation, safety backups, and validation tests for both deployment modes - Created config/backup-cron.sh: installer for daily/weekly automated backup cron jobs (2 AM daily, 3 AM Sunday) All documentation covers dual-deployment modes with shared configuration files. Documentation is operator-ready with copy-paste commands and clear expected outputs.
This commit is contained in:
436
docs/EMERGENCY_PROCEDURES.md
Normal file
436
docs/EMERGENCY_PROCEDURES.md
Normal file
@@ -0,0 +1,436 @@
|
||||
# Emergency Procedures
|
||||
|
||||
**Purpose**: Quick reference for critical incident response.
|
||||
**Audience**: On-call operations team
|
||||
**Response Time Goal**: <5 minutes to action, <10 minutes to recovery
|
||||
|
||||
---
|
||||
|
||||
## Quick Response Matrix
|
||||
|
||||
| Issue | Detection | Immediate Action | Recovery Time |
|
||||
|-------|-----------|------------------|-----------------|
|
||||
| **Service Down** | Ping fails / curl fails | Restart service | 2-3 min |
|
||||
| **API Unresponsive** | /health returns error | Restart backend | 3-5 min |
|
||||
| **Database Locked** | "Database is locked" error | Restart backend | 3-5 min |
|
||||
| **High Memory** | `docker stats` >80% | Kill & restart | 5 min |
|
||||
| **Disk Full** | `df -h` >90% | Clean backups | 5 min |
|
||||
| **Data Corruption** | Integrity check fails | Restore backup | 8-10 min |
|
||||
|
||||
---
|
||||
|
||||
## Emergency Response Playbook
|
||||
|
||||
### INCIDENT 1: Service Down (10 min recovery target)
|
||||
|
||||
**Detection**: `curl http://localhost:8000/health` returns nothing or "connection refused"
|
||||
|
||||
**Immediate (30 seconds)**:
|
||||
```bash
|
||||
# Check service status
|
||||
docker-compose ps # Docker mode
|
||||
ps aux | grep uvicorn # Standalone mode
|
||||
|
||||
# Check if port is actually in use
|
||||
netstat -tuln | grep 8000
|
||||
```
|
||||
|
||||
**Diagnosis (1 minute)**:
|
||||
```bash
|
||||
# Docker mode
|
||||
docker-compose logs backend | tail -50
|
||||
|
||||
# Standalone mode
|
||||
tail -50 logs/backend.log
|
||||
```
|
||||
|
||||
**Recovery (Docker, <3 minutes)**:
|
||||
```bash
|
||||
# Option 1: Restart service
|
||||
docker-compose restart backend
|
||||
|
||||
# Option 2: Full restart (if restart fails)
|
||||
docker-compose down
|
||||
docker-compose up -d
|
||||
|
||||
# Option 3: Emergency (hard reset)
|
||||
docker-compose down
|
||||
rm -f data/inventory.db-* # Remove lock files
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
**Recovery (Standalone, <3 minutes)**:
|
||||
```bash
|
||||
# Kill process
|
||||
pkill -9 -f uvicorn
|
||||
|
||||
# Wait for port to release
|
||||
sleep 3
|
||||
|
||||
# Restart service
|
||||
cd backend && source venv/bin/activate && \
|
||||
uvicorn main:app --host 0.0.0.0 --port 8000 &
|
||||
```
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
curl -v http://localhost:8000/health
|
||||
# Expected: HTTP 200 OK, response time <100ms
|
||||
```
|
||||
|
||||
**Escalate if**: Still not responsive after 5 minutes → Check logs → Call developer support
|
||||
|
||||
---
|
||||
|
||||
### INCIDENT 2: Database Locked (5 min recovery target)
|
||||
|
||||
**Detection**: Requests returning "database is locked" errors
|
||||
|
||||
**Immediate (30 seconds)**:
|
||||
```bash
|
||||
# Docker mode
|
||||
docker-compose logs backend | grep -i "locked" | tail -10
|
||||
|
||||
# Standalone mode
|
||||
tail -20 logs/backend.log | grep -i "locked"
|
||||
```
|
||||
|
||||
**Recovery**:
|
||||
```bash
|
||||
# Docker mode
|
||||
docker-compose restart backend
|
||||
|
||||
# Standalone mode
|
||||
pkill -9 -f uvicorn
|
||||
sleep 2
|
||||
./start_server.sh
|
||||
```
|
||||
|
||||
**Verify**:
|
||||
```bash
|
||||
curl http://localhost:8000/health
|
||||
# Should respond with 200 OK
|
||||
```
|
||||
|
||||
**If still failing**: Restore from backup → See INCIDENT 5
|
||||
|
||||
---
|
||||
|
||||
### INCIDENT 3: High CPU/Memory (5 min recovery target)
|
||||
|
||||
**Detection**: `docker stats` shows >70% CPU or >500MB RAM for backend
|
||||
|
||||
**Immediate (30 seconds)**:
|
||||
```bash
|
||||
# Check resource usage
|
||||
docker stats --no-stream # Docker
|
||||
ps aux | grep uvicorn # Standalone
|
||||
|
||||
# Kill slow query (if identifiable)
|
||||
docker-compose logs backend | grep "slow" | tail -5
|
||||
```
|
||||
|
||||
**Recovery**:
|
||||
```bash
|
||||
# Option 1: Restart service
|
||||
docker-compose restart backend # Docker
|
||||
pkill -f uvicorn # Standalone
|
||||
|
||||
# Option 2: Limit resources (Docker only)
|
||||
# Edit docker-compose.yml:
|
||||
# backend:
|
||||
# mem_limit: 1g
|
||||
|
||||
# Option 3: Investigate slow queries
|
||||
docker-compose exec backend python -c "
|
||||
import backend.models
|
||||
from backend.db import SessionLocal
|
||||
db = SessionLocal()
|
||||
# Run diagnostic queries
|
||||
"
|
||||
```
|
||||
|
||||
**Monitor**: Watch for 10 minutes after restart to ensure stable
|
||||
|
||||
---
|
||||
|
||||
### INCIDENT 4: Disk Full (5 min recovery target)
|
||||
|
||||
**Detection**: `df -h` shows 90%+ usage, write operations failing
|
||||
|
||||
**Immediate (1 minute)**:
|
||||
```bash
|
||||
# Check disk usage
|
||||
du -sh /* | sort -rh | head -10
|
||||
|
||||
# Identify largest items
|
||||
du -sh data/ backups/ logs/
|
||||
```
|
||||
|
||||
**Recovery (order of priority)**:
|
||||
```bash
|
||||
# 1. Delete old backups (usually >90% of disk)
|
||||
find backups/ -name "inventory-*.tar.gz" -mtime +7 -delete
|
||||
# Safe: Backups older than 7 days
|
||||
# Aggressive: -mtime +3 (3 days)
|
||||
|
||||
# 2. Compress old logs
|
||||
gzip logs/*.log.* 2>/dev/null || true
|
||||
find logs/ -name "*.gz" -mtime +30 -delete
|
||||
|
||||
# 3. Vacuum database (if >500MB)
|
||||
sqlite3 data/inventory.db "VACUUM;"
|
||||
|
||||
# 4. Delete oldest backups if still full
|
||||
find backups/ -name "*.tar.gz" -type f | sort | head -1 | xargs rm
|
||||
```
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
df -h # Should be <80% now
|
||||
du -sh backups/
|
||||
```
|
||||
|
||||
**Prevention**: Increase disk size or set up offsite backups
|
||||
|
||||
---
|
||||
|
||||
### INCIDENT 5: Data Corruption (10 min recovery target)
|
||||
|
||||
**Detection**: Database integrity check fails, unexpected data missing, query errors
|
||||
|
||||
**Immediate (1 minute)**:
|
||||
```bash
|
||||
# Verify corruption
|
||||
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA integrity_check;"
|
||||
# OR (Standalone)
|
||||
sqlite3 data/inventory.db "PRAGMA integrity_check;"
|
||||
|
||||
# Check logs for errors
|
||||
docker-compose logs backend | grep -i "error" | tail -20
|
||||
```
|
||||
|
||||
**Recovery (8-10 minutes)**:
|
||||
```bash
|
||||
# CRITICAL: Do not attempt to repair
|
||||
# Restore from backup (fastest, safest option)
|
||||
|
||||
# 1. Check available backups
|
||||
ls -lh backups/ | head -5
|
||||
|
||||
# 2. Stop services
|
||||
docker-compose down # Docker
|
||||
pkill -f uvicorn # Standalone
|
||||
|
||||
# 3. Restore
|
||||
./scripts/restore.sh backups/latest.tar.gz --validate
|
||||
|
||||
# 4. Restart (if needed)
|
||||
docker-compose up -d # Docker
|
||||
./start_server.sh # Standalone
|
||||
|
||||
# 5. Verify
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
**Notify Users**: Data loss = up to 1 day (latest backup)
|
||||
|
||||
**Escalate**: Call database admin after recovery
|
||||
|
||||
---
|
||||
|
||||
### INCIDENT 6: Network / CORS Errors (5 min recovery target)
|
||||
|
||||
**Detection**: Browser console shows CORS error, frontend can't reach backend
|
||||
|
||||
**Immediate (1 minute)**:
|
||||
```bash
|
||||
# Test backend connectivity
|
||||
curl -v http://localhost:8000/health
|
||||
curl -v http://<server-ip>:8000/health
|
||||
|
||||
# Check CORS configuration
|
||||
docker-compose exec backend python -c "
|
||||
from backend.config import ALLOWED_ORIGINS
|
||||
print('ALLOWED_ORIGINS:', ALLOWED_ORIGINS)
|
||||
"
|
||||
```
|
||||
|
||||
**Recovery**:
|
||||
```bash
|
||||
# 1. Check network connectivity
|
||||
ping <backend-server-ip>
|
||||
|
||||
# 2. Update CORS if needed
|
||||
# Edit inventory.env:
|
||||
EXTRA_ALLOWED_ORIGINS=<client-ip>
|
||||
|
||||
# 3. Restart backend
|
||||
docker-compose restart backend
|
||||
|
||||
# 4. Test
|
||||
curl -H "Origin: http://<client-ip>" -v http://localhost:8000/health
|
||||
```
|
||||
|
||||
**Verify**: Frontend should load without CORS errors
|
||||
|
||||
---
|
||||
|
||||
### INCIDENT 7: Frontend Not Loading (5 min recovery target)
|
||||
|
||||
**Detection**: Frontend port doesn't respond, blank page, 404 errors
|
||||
|
||||
**Recovery (Docker)**:
|
||||
```bash
|
||||
# Check service
|
||||
docker-compose ps | grep frontend
|
||||
|
||||
# Restart
|
||||
docker-compose restart frontend
|
||||
|
||||
# Check logs
|
||||
docker-compose logs frontend | tail -50
|
||||
|
||||
# If build failed, rebuild
|
||||
docker-compose down
|
||||
docker-compose up -d --build
|
||||
```
|
||||
|
||||
**Recovery (Standalone)**:
|
||||
```bash
|
||||
# Kill process
|
||||
pkill -f "next start"
|
||||
|
||||
# Rebuild if needed
|
||||
cd frontend && npm install && npm run build
|
||||
|
||||
# Restart
|
||||
cd .. && npm start --prefix frontend &
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Emergency Decision Tree
|
||||
|
||||
```
|
||||
Service not responding?
|
||||
├─ YES: INCIDENT 1 (Service Down)
|
||||
└─ NO: Continue
|
||||
|
||||
Getting "locked" errors?
|
||||
├─ YES: INCIDENT 2 (Database Locked)
|
||||
└─ NO: Continue
|
||||
|
||||
High CPU/Memory?
|
||||
├─ YES: INCIDENT 3 (High Resources)
|
||||
└─ NO: Continue
|
||||
|
||||
Disk full?
|
||||
├─ YES: INCIDENT 4 (Disk Full)
|
||||
└─ NO: Continue
|
||||
|
||||
Data missing/corrupted?
|
||||
├─ YES: INCIDENT 5 (Data Corruption)
|
||||
└─ NO: Continue
|
||||
|
||||
CORS/Network errors?
|
||||
├─ YES: INCIDENT 6 (Network Issues)
|
||||
└─ NO: Continue
|
||||
|
||||
Frontend not loading?
|
||||
├─ YES: INCIDENT 7 (Frontend Error)
|
||||
└─ NO: Contact developer support
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Escalation Path
|
||||
|
||||
### Tier 1: On-Call Operations (You are here)
|
||||
- [ ] Attempt immediate recovery (restart, clear locks)
|
||||
- [ ] Document issue and time
|
||||
- [ ] If not resolved in 5 minutes → Escalate
|
||||
|
||||
### Tier 2: Senior DevOps / Backup On-Call
|
||||
- [ ] Call: [Phone]
|
||||
- [ ] Message: "TFM Inventory [INCIDENT]: [Description]"
|
||||
- [ ] Provide: Error messages, logs, recovery attempts
|
||||
|
||||
### Tier 3: Application Developer
|
||||
- [ ] If Tier 2 unresponsive for 10 minutes
|
||||
- [ ] Call: [Phone]
|
||||
- [ ] Include: Full logs, screenshots
|
||||
|
||||
### Tier 4: Management
|
||||
- [ ] If service down >30 minutes
|
||||
- [ ] Notify: [Manager], [Director]
|
||||
|
||||
---
|
||||
|
||||
## Post-Incident Actions
|
||||
|
||||
**Within 1 hour**:
|
||||
- [ ] Document issue and resolution
|
||||
- [ ] Note start time, detection time, resolution time
|
||||
- [ ] Save error logs to archive
|
||||
|
||||
**Within 24 hours**:
|
||||
- [ ] Root cause analysis
|
||||
- [ ] Identify prevention measures
|
||||
- [ ] Update runbooks if needed
|
||||
|
||||
**Within 1 week**:
|
||||
- [ ] Implement preventive fix
|
||||
- [ ] Update monitoring rules
|
||||
- [ ] Run incident review with team
|
||||
|
||||
---
|
||||
|
||||
## Critical Contacts
|
||||
|
||||
| Role | Name | Phone | Email |
|
||||
|------|------|-------|-------|
|
||||
| On-Call Ops | [Name] | [+1-xxx-xxx-xxxx] | [Email] |
|
||||
| Backup Ops | [Name] | [+1-xxx-xxx-xxxx] | [Email] |
|
||||
| Senior DevOps | [Name] | [+1-xxx-xxx-xxxx] | [Email] |
|
||||
| Developer | [Name] | [+1-xxx-xxx-xxxx] | [Email] |
|
||||
|
||||
---
|
||||
|
||||
## Cheat Sheet (Print and Post)
|
||||
|
||||
```
|
||||
QUICK FIXES:
|
||||
|
||||
Service Down?
|
||||
docker-compose restart backend
|
||||
|
||||
Database Locked?
|
||||
docker-compose restart backend
|
||||
|
||||
Disk Full?
|
||||
find backups/ -name "*.tar.gz" -mtime +7 -delete
|
||||
|
||||
Data Corrupted?
|
||||
./scripts/restore.sh backups/latest.tar.gz --validate
|
||||
|
||||
CORS Error?
|
||||
Edit inventory.env + docker-compose restart backend
|
||||
|
||||
Check Health:
|
||||
curl http://localhost:8000/health
|
||||
|
||||
View Logs:
|
||||
docker-compose logs -f backend
|
||||
|
||||
CONTACTS:
|
||||
On-call: [Phone]
|
||||
Dev Support: [Email]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Version**: 1.0
|
||||
**Last Updated**: 2026-04-22
|
||||
**Next Review**: 2026-05-22
|
||||
**Owner**: Operations Team
|
||||
Reference in New Issue
Block a user