tfm_ainventory/docs/EMERGENCY_PROCEDURES.md

Emergency Procedures

Purpose: Quick reference for critical incident response.
Audience: On-call operations team
Response Time Goal: <5 minutes to action, <10 minutes to recovery

---

Quick Response Matrix

Issue	Detection	Immediate Action	Recovery Time
Service Down	Ping fails / curl fails	Restart service	2-3 min
API Unresponsive	/health returns error	Restart backend	3-5 min
Database Locked	"Database is locked" error	Restart backend	3-5 min
High Memory	docker stats >80%	Kill & restart	5 min
Disk Full	df -h >90%	Clean backups	5 min
Data Corruption	Integrity check fails	Restore backup	8-10 min

---

Emergency Response Playbook

INCIDENT 1: Service Down (10 min recovery target)

Detection: curl http://localhost:8000/health returns nothing or "connection refused"

Immediate (30 seconds):

# Check service status
docker-compose ps  # Docker mode
ps aux | grep uvicorn  # Standalone mode

# Check if port is actually in use
netstat -tuln | grep 8000

Diagnosis (1 minute):

# Docker mode
docker-compose logs backend | tail -50

# Standalone mode
tail -50 logs/backend.log

Recovery (Docker, <3 minutes):

# Option 1: Restart service
docker-compose restart backend

# Option 2: Full restart (if restart fails)
docker-compose down
docker-compose up -d

# Option 3: Emergency (hard reset)
docker-compose down
rm -f data/inventory.db-*  # Remove lock files
docker-compose up -d

Recovery (Standalone, <3 minutes):

# Kill process
pkill -9 -f uvicorn

# Wait for port to release
sleep 3

# Restart service
cd backend && source venv/bin/activate && \
  uvicorn main:app --host 0.0.0.0 --port 8000 &

Verification:

curl -v http://localhost:8000/health
# Expected: HTTP 200 OK, response time <100ms

Escalate if: Still not responsive after 5 minutes → Check logs → Call developer support

---

INCIDENT 2: Database Locked (5 min recovery target)

Detection: Requests returning "database is locked" errors

Immediate (30 seconds):

# Docker mode
docker-compose logs backend | grep -i "locked" | tail -10

# Standalone mode
tail -20 logs/backend.log | grep -i "locked"

Recovery:

# Docker mode
docker-compose restart backend

# Standalone mode
pkill -9 -f uvicorn
sleep 2
./start_server.sh

Verify:

curl http://localhost:8000/health
# Should respond with 200 OK

If still failing: Restore from backup → See INCIDENT 5

---

INCIDENT 3: High CPU/Memory (5 min recovery target)

Detection: docker stats shows >70% CPU or >500MB RAM for backend

Immediate (30 seconds):

# Check resource usage
docker stats --no-stream  # Docker
ps aux | grep uvicorn  # Standalone

# Kill slow query (if identifiable)
docker-compose logs backend | grep "slow" | tail -5

Recovery:

# Option 1: Restart service
docker-compose restart backend  # Docker
pkill -f uvicorn  # Standalone

# Option 2: Limit resources (Docker only)
# Edit docker-compose.yml:
# backend:
#   mem_limit: 1g

# Option 3: Investigate slow queries
docker-compose exec backend python -c "
import backend.models
from backend.db import SessionLocal
db = SessionLocal()
# Run diagnostic queries
"

Monitor: Watch for 10 minutes after restart to ensure stable

---

INCIDENT 4: Disk Full (5 min recovery target)

Detection: df -h shows 90%+ usage, write operations failing

Immediate (1 minute):

# Check disk usage
du -sh /* | sort -rh | head -10

# Identify largest items
du -sh data/ backups/ logs/

Recovery (order of priority):

# 1. Delete old backups (usually >90% of disk)
find backups/ -name "inventory-*.tar.gz" -mtime +7 -delete
# Safe: Backups older than 7 days
# Aggressive: -mtime +3 (3 days)

# 2. Compress old logs
gzip logs/*.log.* 2>/dev/null || true
find logs/ -name "*.gz" -mtime +30 -delete

# 3. Vacuum database (if >500MB)
sqlite3 data/inventory.db "VACUUM;"

# 4. Delete oldest backups if still full
find backups/ -name "*.tar.gz" -type f | sort | head -1 | xargs rm

Verification:

df -h  # Should be <80% now
du -sh backups/

Prevention: Increase disk size or set up offsite backups

---

INCIDENT 5: Data Corruption (10 min recovery target)

Detection: Database integrity check fails, unexpected data missing, query errors

Immediate (1 minute):

# Verify corruption
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA integrity_check;"
# OR (Standalone)
sqlite3 data/inventory.db "PRAGMA integrity_check;"

# Check logs for errors
docker-compose logs backend | grep -i "error" | tail -20

Recovery (8-10 minutes):

# CRITICAL: Do not attempt to repair
# Restore from backup (fastest, safest option)

# 1. Check available backups
ls -lh backups/ | head -5

# 2. Stop services
docker-compose down  # Docker
pkill -f uvicorn  # Standalone

# 3. Restore
./scripts/restore.sh backups/latest.tar.gz --validate

# 4. Restart (if needed)
docker-compose up -d  # Docker
./start_server.sh  # Standalone

# 5. Verify
curl http://localhost:8000/health

Notify Users: Data loss = up to 1 day (latest backup)

Escalate: Call database admin after recovery

---

INCIDENT 6: Network / CORS Errors (5 min recovery target)

Detection: Browser console shows CORS error, frontend can't reach backend

Immediate (1 minute):

# Test backend connectivity
curl -v http://localhost:8000/health
curl -v http://<server-ip>:8000/health

# Check CORS configuration
docker-compose exec backend python -c "
from backend.config import ALLOWED_ORIGINS
print('ALLOWED_ORIGINS:', ALLOWED_ORIGINS)
"

Recovery:

# 1. Check network connectivity
ping <backend-server-ip>

# 2. Update CORS if needed
# Edit inventory.env:
EXTRA_ALLOWED_ORIGINS=<client-ip>

# 3. Restart backend
docker-compose restart backend

# 4. Test
curl -H "Origin: http://<client-ip>" -v http://localhost:8000/health

Verify: Frontend should load without CORS errors

---

INCIDENT 7: Frontend Not Loading (5 min recovery target)

Detection: Frontend port doesn't respond, blank page, 404 errors

Recovery (Docker):

# Check service
docker-compose ps | grep frontend

# Restart
docker-compose restart frontend

# Check logs
docker-compose logs frontend | tail -50

# If build failed, rebuild
docker-compose down
docker-compose up -d --build

Recovery (Standalone):

# Kill process
pkill -f "next start"

# Rebuild if needed
cd frontend && npm install && npm run build

# Restart
cd .. && npm start --prefix frontend &

---

Emergency Decision Tree

Service not responding?
├─ YES: INCIDENT 1 (Service Down)
└─ NO: Continue

Getting "locked" errors?
├─ YES: INCIDENT 2 (Database Locked)
└─ NO: Continue

High CPU/Memory?
├─ YES: INCIDENT 3 (High Resources)
└─ NO: Continue

Disk full?
├─ YES: INCIDENT 4 (Disk Full)
└─ NO: Continue

Data missing/corrupted?
├─ YES: INCIDENT 5 (Data Corruption)
└─ NO: Continue

CORS/Network errors?
├─ YES: INCIDENT 6 (Network Issues)
└─ NO: Continue

Frontend not loading?
├─ YES: INCIDENT 7 (Frontend Error)
└─ NO: Contact developer support

---

Escalation Path

Tier 1: On-Call Operations (You are here)

• Attempt immediate recovery (restart, clear locks)
• Document issue and time
• If not resolved in 5 minutes → Escalate

Tier 2: Senior DevOps / Backup On-Call

• Call: [Phone]
• Message: "TFM Inventory [INCIDENT]: [Description]"
• Provide: Error messages, logs, recovery attempts

Tier 3: Application Developer

• If Tier 2 unresponsive for 10 minutes
• Call: [Phone]
• Include: Full logs, screenshots

Tier 4: Management

• If service down >30 minutes
• Notify: [Manager], [Director]

---

Post-Incident Actions

Within 1 hour:

• Document issue and resolution
• Note start time, detection time, resolution time
• Save error logs to archive

Within 24 hours:

• Root cause analysis
• Identify prevention measures
• Update runbooks if needed

Within 1 week:

• Implement preventive fix
• Update monitoring rules
• Run incident review with team

---

Critical Contacts

Role	Name	Phone	Email
On-Call Ops	[Name]	[+1-xxx-xxx-xxxx]	[Email]
Backup Ops	[Name]	[+1-xxx-xxx-xxxx]	[Email]
Senior DevOps	[Name]	[+1-xxx-xxx-xxxx]	[Email]
Developer	[Name]	[+1-xxx-xxx-xxxx]	[Email]

---

Cheat Sheet (Print and Post)

QUICK FIXES:

Service Down?
  docker-compose restart backend

Database Locked?
  docker-compose restart backend

Disk Full?
  find backups/ -name "*.tar.gz" -mtime +7 -delete

Data Corrupted?
  ./scripts/restore.sh backups/latest.tar.gz --validate

CORS Error?
  Edit inventory.env + docker-compose restart backend

Check Health:
  curl http://localhost:8000/health

View Logs:
  docker-compose logs -f backend

CONTACTS:
  On-call: [Phone]
  Dev Support: [Email]

---

Version: 1.0
Last Updated: 2026-04-22
Next Review: 2026-05-22
Owner: Operations Team