tfm_ainventory/docs/EMERGENCY_PROCEDURES.md

# Emergency Procedures

**Purpose**: Quick reference for critical incident response.  
**Audience**: On-call operations team  
**Response Time Goal**: <5 minutes to action, <10 minutes to recovery

---

## Quick Response Matrix

| Issue | Detection | Immediate Action | Recovery Time |
|-------|-----------|------------------|-----------------|
| **Service Down** | Ping fails / curl fails | Restart service | 2-3 min |
| **API Unresponsive** | /health returns error | Restart backend | 3-5 min |
| **Database Locked** | "Database is locked" error | Restart backend | 3-5 min |
| **High Memory** | `docker stats` >80% | Kill & restart | 5 min |
| **Disk Full** | `df -h` >90% | Clean backups | 5 min |
| **Data Corruption** | Integrity check fails | Restore backup | 8-10 min |

---

## Emergency Response Playbook

### INCIDENT 1: Service Down (10 min recovery target)

**Detection**: `curl http://localhost:8000/health` returns nothing or "connection refused"

**Immediate (30 seconds)**:
```bash
# Check service status
docker-compose ps  # Docker mode
ps aux | grep uvicorn  # Standalone mode

# Check if port is actually in use
netstat -tuln | grep 8000
```

**Diagnosis (1 minute)**:
```bash
# Docker mode
docker-compose logs backend | tail -50

# Standalone mode
tail -50 logs/backend.log
```

**Recovery (Docker, <3 minutes)**:
```bash
# Option 1: Restart service
docker-compose restart backend

# Option 2: Full restart (if restart fails)
docker-compose down
docker-compose up -d

# Option 3: Emergency (hard reset)
docker-compose down
rm -f data/inventory.db-*  # Remove lock files
docker-compose up -d
```

**Recovery (Standalone, <3 minutes)**:
```bash
# Kill process
pkill -9 -f uvicorn

# Wait for port to release
sleep 3

# Restart service
cd backend && source venv/bin/activate && \
  uvicorn main:app --host 0.0.0.0 --port 8000 &
```

**Verification**:
```bash
curl -v http://localhost:8000/health
# Expected: HTTP 200 OK, response time <100ms
```

**Escalate if**: Still not responsive after 5 minutes → Check logs → Call developer support

---

### INCIDENT 2: Database Locked (5 min recovery target)

**Detection**: Requests returning "database is locked" errors

**Immediate (30 seconds)**:
```bash
# Docker mode
docker-compose logs backend | grep -i "locked" | tail -10

# Standalone mode
tail -20 logs/backend.log | grep -i "locked"
```

**Recovery**:
```bash
# Docker mode
docker-compose restart backend

# Standalone mode
pkill -9 -f uvicorn
sleep 2
./start_server.sh
```

**Verify**:
```bash
curl http://localhost:8000/health
# Should respond with 200 OK
```

**If still failing**: Restore from backup → See INCIDENT 5

---

### INCIDENT 3: High CPU/Memory (5 min recovery target)

**Detection**: `docker stats` shows >70% CPU or >500MB RAM for backend

**Immediate (30 seconds)**:
```bash
# Check resource usage
docker stats --no-stream  # Docker
ps aux | grep uvicorn  # Standalone

# Kill slow query (if identifiable)
docker-compose logs backend | grep "slow" | tail -5
```

**Recovery**:
```bash
# Option 1: Restart service
docker-compose restart backend  # Docker
pkill -f uvicorn  # Standalone

# Option 2: Limit resources (Docker only)
# Edit docker-compose.yml:
# backend:
#   mem_limit: 1g

# Option 3: Investigate slow queries
docker-compose exec backend python -c "
import backend.models
from backend.db import SessionLocal
db = SessionLocal()
# Run diagnostic queries
"
```

**Monitor**: Watch for 10 minutes after restart to ensure stable

---

### INCIDENT 4: Disk Full (5 min recovery target)

**Detection**: `df -h` shows 90%+ usage, write operations failing

**Immediate (1 minute)**:
```bash
# Check disk usage
du -sh /* | sort -rh | head -10

# Identify largest items
du -sh data/ backups/ logs/
```

**Recovery (order of priority)**:
```bash
# 1. Delete old backups (usually >90% of disk)
find backups/ -name "inventory-*.tar.gz" -mtime +7 -delete
# Safe: Backups older than 7 days
# Aggressive: -mtime +3 (3 days)

# 2. Compress old logs
gzip logs/*.log.* 2>/dev/null || true
find logs/ -name "*.gz" -mtime +30 -delete

# 3. Vacuum database (if >500MB)
sqlite3 data/inventory.db "VACUUM;"

# 4. Delete oldest backups if still full
find backups/ -name "*.tar.gz" -type f | sort | head -1 | xargs rm
```

**Verification**:
```bash
df -h  # Should be <80% now
du -sh backups/
```

**Prevention**: Increase disk size or set up offsite backups

---

### INCIDENT 5: Data Corruption (10 min recovery target)

**Detection**: Database integrity check fails, unexpected data missing, query errors

**Immediate (1 minute)**:
```bash
# Verify corruption
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA integrity_check;"
# OR (Standalone)
sqlite3 data/inventory.db "PRAGMA integrity_check;"

# Check logs for errors
docker-compose logs backend | grep -i "error" | tail -20
```

**Recovery (8-10 minutes)**:
```bash
# CRITICAL: Do not attempt to repair
# Restore from backup (fastest, safest option)

# 1. Check available backups
ls -lh backups/ | head -5

# 2. Stop services
docker-compose down  # Docker
pkill -f uvicorn  # Standalone

# 3. Restore
./scripts/restore.sh backups/latest.tar.gz --validate

# 4. Restart (if needed)
docker-compose up -d  # Docker
./start_server.sh  # Standalone

# 5. Verify
curl http://localhost:8000/health
```

**Notify Users**: Data loss = up to 1 day (latest backup)

**Escalate**: Call database admin after recovery

---

### INCIDENT 6: Network / CORS Errors (5 min recovery target)

**Detection**: Browser console shows CORS error, frontend can't reach backend

**Immediate (1 minute)**:
```bash
# Test backend connectivity
curl -v http://localhost:8000/health
curl -v http://<server-ip>:8000/health

# Check CORS configuration
docker-compose exec backend python -c "
from backend.config import ALLOWED_ORIGINS
print('ALLOWED_ORIGINS:', ALLOWED_ORIGINS)
"
```

**Recovery**:
```bash
# 1. Check network connectivity
ping <backend-server-ip>

# 2. Update CORS if needed
# Edit inventory.env:
EXTRA_ALLOWED_ORIGINS=<client-ip>

# 3. Restart backend
docker-compose restart backend

# 4. Test
curl -H "Origin: http://<client-ip>" -v http://localhost:8000/health
```

**Verify**: Frontend should load without CORS errors

---

### INCIDENT 7: Frontend Not Loading (5 min recovery target)

**Detection**: Frontend port doesn't respond, blank page, 404 errors

**Recovery (Docker)**:
```bash
# Check service
docker-compose ps | grep frontend

# Restart
docker-compose restart frontend

# Check logs
docker-compose logs frontend | tail -50

# If build failed, rebuild
docker-compose down
docker-compose up -d --build
```

**Recovery (Standalone)**:
```bash
# Kill process
pkill -f "next start"

# Rebuild if needed
cd frontend && npm install && npm run build

# Restart
cd .. && npm start --prefix frontend &
```

---

## Emergency Decision Tree

```
Service not responding?
├─ YES: INCIDENT 1 (Service Down)
└─ NO: Continue

Getting "locked" errors?
├─ YES: INCIDENT 2 (Database Locked)
└─ NO: Continue

High CPU/Memory?
├─ YES: INCIDENT 3 (High Resources)
└─ NO: Continue

Disk full?
├─ YES: INCIDENT 4 (Disk Full)
└─ NO: Continue

Data missing/corrupted?
├─ YES: INCIDENT 5 (Data Corruption)
└─ NO: Continue

CORS/Network errors?
├─ YES: INCIDENT 6 (Network Issues)
└─ NO: Continue

Frontend not loading?
├─ YES: INCIDENT 7 (Frontend Error)
└─ NO: Contact developer support
```

---

## Escalation Path

### Tier 1: On-Call Operations (You are here)
- [ ] Attempt immediate recovery (restart, clear locks)
- [ ] Document issue and time
- [ ] If not resolved in 5 minutes → Escalate

### Tier 2: Senior DevOps / Backup On-Call
- [ ] Call: [Phone]
- [ ] Message: "TFM Inventory [INCIDENT]: [Description]"
- [ ] Provide: Error messages, logs, recovery attempts

### Tier 3: Application Developer
- [ ] If Tier 2 unresponsive for 10 minutes
- [ ] Call: [Phone]
- [ ] Include: Full logs, screenshots

### Tier 4: Management
- [ ] If service down >30 minutes
- [ ] Notify: [Manager], [Director]

---

## Post-Incident Actions

**Within 1 hour**:
- [ ] Document issue and resolution
- [ ] Note start time, detection time, resolution time
- [ ] Save error logs to archive

**Within 24 hours**:
- [ ] Root cause analysis
- [ ] Identify prevention measures
- [ ] Update runbooks if needed

**Within 1 week**:
- [ ] Implement preventive fix
- [ ] Update monitoring rules
- [ ] Run incident review with team

---

## Critical Contacts

| Role | Name | Phone | Email |
|------|------|-------|-------|
| On-Call Ops | [Name] | [+1-xxx-xxx-xxxx] | [Email] |
| Backup Ops | [Name] | [+1-xxx-xxx-xxxx] | [Email] |
| Senior DevOps | [Name] | [+1-xxx-xxx-xxxx] | [Email] |
| Developer | [Name] | [+1-xxx-xxx-xxxx] | [Email] |

---

## Cheat Sheet (Print and Post)

```
QUICK FIXES:

Service Down?
  docker-compose restart backend

Database Locked?
  docker-compose restart backend

Disk Full?
  find backups/ -name "*.tar.gz" -mtime +7 -delete

Data Corrupted?
  ./scripts/restore.sh backups/latest.tar.gz --validate

CORS Error?
  Edit inventory.env + docker-compose restart backend

Check Health:
  curl http://localhost:8000/health

View Logs:
  docker-compose logs -f backend

CONTACTS:
  On-call: [Phone]
  Dev Support: [Email]
```

---

**Version**: 1.0  
**Last Updated**: 2026-04-22  
**Next Review**: 2026-05-22  
**Owner**: Operations Team