tfm_ainventory/docs/DISASTER_RECOVERY_PLAN.md

Disaster Recovery Plan

Objective: Restore production service within 10 minutes and zero data loss.
Status: Active
Last Tested: [Date]
Next Review: 2026-05-22

---

Overview

This document outlines procedures for recovering from various failure scenarios. The system uses automated daily backups with a 1-day RPO (Recovery Point Objective) and aims for <10 minute RTO (Recovery Time Objective).

---

Scenarios & Recovery Procedures

Scenario 1: Database Corrupted

Detection:

• Integrity check fails: PRAGMA integrity_check;
• Data unexpectedly missing
• Queries returning errors

Recovery Steps (Docker):

# 1. Verify corruption
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA integrity_check;"

# 2. Stop services
docker-compose down

# 3. Restore from backup
./scripts/restore.sh backups/latest.tar.gz --validate

# 4. Verify data integrity
docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"

Recovery Steps (Standalone):

# 1. Verify corruption
sqlite3 data/inventory.db "PRAGMA integrity_check;"

# 2. Stop services
pkill -f uvicorn
pkill -f "next start"

# 3. Restore from backup
./scripts/restore.sh backups/latest.tar.gz

# 4. Start services
./start_server.sh

# 5. Verify
sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"

RTO: <10 minutes
RPO: 1 day
Notify Users: If data loss within last 24 hours

---

Scenario 2: Complete Hardware Failure

Detection:

• Server doesn't boot
• Server not reachable on network
• Docker daemon won't start

Recovery Steps:

# 1. Provision new Ubuntu 22.04 LTS server
#    Same specs as original (2GB+ RAM, 10GB+ disk)

# 2. Clone repository
git clone <repo_url> /opt/tfm-inventory
cd /opt/tfm-inventory

# 3. Copy backup from offsite storage
#    (Assuming you have offsite backup copy)
cp /path/to/offsite/backup-latest.tar.gz ./backups/

# 4. Restore
./scripts/restore.sh backups/backup-latest.tar.gz --validate

# 5. Update DNS/load balancer to new IP

# 6. Verify services
curl http://localhost:8000/health
curl http://localhost:3000

RTO: <30 minutes (depends on provisioning speed)
RPO: 1 day
Estimated Cost: New hardware provisioning

---

Scenario 3: Data Center Failure

Detection:

• Entire data center unreachable
• Multiple systems down simultaneously
• Network infrastructure down

Recovery Steps:

# 1. Activate secondary site (if available)
#    or failover to cloud provider

# 2. Provision new infrastructure
#    Clone repository on new infrastructure

# 3. Restore latest backup
git clone <repo> /opt/tfm-inventory
cd /opt/tfm-inventory
./scripts/restore.sh /offsite/backup-latest.tar.gz --validate

# 4. Update DNS to new location
#    (Allow 5-15 min for DNS propagation)

# 5. Notify users
#    "Service restored; data loss = last 1 day"

RTO: 30-60 minutes (depends on secondary readiness)
RPO: 1 day
Prevention: Maintain offsite backup copy at minimum

---

Scenario 4: Application Crash / Memory Leak

Detection:

• Backend crashes and won't restart
• Frontend crashes
• Memory continuously growing

Recovery Steps:

# Docker mode:
docker-compose logs backend | tail -100

# If memory leak:
docker-compose restart backend

# If crash persists:
git log --oneline | head -10
git revert <commit-hash>
./deploy.sh production

# Standalone mode:
tail -100 logs/backend.log

pkill -9 -f uvicorn
./start_server.sh

# If crash persists:
git revert <problematic-commit>
./start_server.sh

RTO: <5 minutes (restart)
RPO: 0 (no data loss, running services)

---

Scenario 5: Disk Full

Detection:

• df -h shows 100% usage
• Write operations failing
• Backup script failing

Recovery Steps:

# 1. Identify large directories
du -sh /* | sort -rh | head -10

# 2. Clean old backups
find backups/ -name "*.tar.gz" -mtime +7 -delete

# 3. Clear logs if very large
find logs/ -name "*.log" -mtime +30 -delete

# 4. Extend disk volume
#    (Depends on cloud provider or physical hardware)

# 5. Verify
df -h

RTO: <15 minutes
RPO: 0 (no data loss)

---

Scenario 6: Network Isolation / CORS Issues

Detection:

• Frontend can't reach backend API
• CORS errors in browser console
• API reachable locally but not from network

Recovery Steps:

# 1. Check network connectivity
ping <backend-ip>
curl -v http://backend-ip:8000/health

# 2. Check CORS configuration
docker-compose exec backend python -c "
from backend.config import ALLOWED_ORIGINS
print(ALLOWED_ORIGINS)
"

# 3. Update CORS if needed
# Edit inventory.env:
# EXTRA_ALLOWED_ORIGINS=<your-ip>

# 4. Restart backend
docker-compose restart backend

# 5. Verify
curl http://localhost:8000/health

RTO: <5 minutes
RPO: 0

---

Regular Testing

Monthly Backup Test

Run on staging environment (not production):

# 1. List available backups
ls -lh backups/

# 2. Restore latest
./scripts/restore.sh backups/latest.tar.gz --validate

# 3. Verify checklist
- [ ] Restore completes without errors
- [ ] All services start correctly
- [ ] Database passes integrity check
- [ ] Item count matches expectation (e.g., 10K+ items)
- [ ] API responds at /health
- [ ] Frontend loads without errors
- [ ] Can login with test account

Quarterly Full Failover Drill

Once per quarter, perform complete failover simulation:

# 1. Provision staging server with identical specs
# 2. Restore production backup
# 3. Run health checklist
# 4. Simulate 5 concurrent users (if load testing available)
# 5. Document any issues
# 6. Update this plan based on findings

Annual Disaster Recovery Exercise

Once per year:

• Simulate data center failure
• Activate secondary site (if exists)
• Full restore on new infrastructure
• Involve all ops team members
• Document timeline and issues
• Update RTO/RPO estimates

---

Prevention & Mitigation

Layer	Prevention	Implementation
Backup	Daily automated	Cron jobs, 30-day rotation
Offsite Backup	Weekly copy to cloud	S3/GCS bucket, encrypted
Monitoring	Alert on issues	CPU >70%, disk >80%, API down
Redundancy	Secondary instance	v3 feature (not in v2 scope)
Testing	Monthly restore drill	Staging environment
Documentation	Up-to-date runbooks	Review quarterly

---

Offsite Backup Setup (Recommended)

To prevent total data loss in case of hardware failure:

# Weekly copy to cloud storage (add to cron)
0 4 * * 0 cd /opt/tfm-inventory && \
  gsutil -m cp backups/inventory-*.tar.gz \
  gs://your-backup-bucket/tfm-inventory/ || \
  aws s3 sync backups/ s3://your-bucket/tfm-inventory/

# Or to another server
0 4 * * 0 cd /opt/tfm-inventory && \
  rsync -avz backups/ backup-server:/backups/tfm-inventory/

---

Communication Plan

During Incident

1. Immediate (notify immediately):

   • CEO / Project Lead
   • Affected users
   • Operations team

2. Message Template:

   Service Status: [DEGRADED|DOWN]
   Impact: Inventory system unavailable
   ETA: <estimated recovery time>
   Action: We are restoring from backup
   

3. Updates: Every 5 minutes or when status changes

After Recovery

1. Post-incident Review: Within 48 hours

   • What failed?
   • Why did it fail?
   • How do we prevent it?
   • Update this plan

2. Root Cause Analysis: Within 1 week

3. Implement Fixes: Within 2 weeks

---

Success Criteria

For recovery to be considered successful:

• Restore completes in <10 minutes (target)
• Zero data loss (max 1 day RPO acceptable)
• All services healthy post-restore
• Users can login and access inventory
• API responds at /health with 200 OK
• Database integrity verified
• Audit logs preserved (immutable)
• Monthly test succeeds 100%

---

Contacts & Escalation

Role	Name	Contact	Hours
On-call Ops	[Name]	[Phone]	24/7
Database Admin	[Name]	[Email]	Business hours
Infrastructure	[Name]	[Email]	Business hours
CEO / Product	[Name]	[Phone]	Escalation only

---

Appendix: Recovery Time Estimates

Scenario	Time	Notes
Restart service	2-3 min	Quick fix for most issues
Restore from backup	8-10 min	DB restore + service startup
New hardware	20-30 min	Provisioning + restore
Data center failover	30-60 min	Depends on secondary readiness
Network reconfiguration	5-15 min	DNS + CORS setup

---

Version: 1.0
Last Updated: 2026-04-22
Last Tested: [Date]
Owner: Operations Team
Next Review: 2026-05-22