# Disaster Recovery Plan

**Objective**: Restore production service within 10 minutes and zero data loss.  
**Status**: Active  
**Last Tested**: [Date]  
**Next Review**: 2026-05-22

---

## Overview

This document outlines procedures for recovering from various failure scenarios. The system uses automated daily backups with a 1-day RPO (Recovery Point Objective) and aims for <10 minute RTO (Recovery Time Objective).

---

## Scenarios & Recovery Procedures

### Scenario 1: Database Corrupted

**Detection**:
- Integrity check fails: `PRAGMA integrity_check;`
- Data unexpectedly missing
- Queries returning errors

**Recovery Steps (Docker)**:
```bash
# 1. Verify corruption
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA integrity_check;"

# 2. Stop services
docker-compose down

# 3. Restore from backup
./scripts/restore.sh backups/latest.tar.gz --validate

# 4. Verify data integrity
docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"
```

**Recovery Steps (Standalone)**:
```bash
# 1. Verify corruption
sqlite3 data/inventory.db "PRAGMA integrity_check;"

# 2. Stop services
pkill -f uvicorn
pkill -f "next start"

# 3. Restore from backup
./scripts/restore.sh backups/latest.tar.gz

# 4. Start services
./start_server.sh

# 5. Verify
sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"
```

**RTO**: <10 minutes  
**RPO**: 1 day  
**Notify Users**: If data loss within last 24 hours

---

### Scenario 2: Complete Hardware Failure

**Detection**:
- Server doesn't boot
- Server not reachable on network
- Docker daemon won't start

**Recovery Steps**:
```bash
# 1. Provision new Ubuntu 22.04 LTS server
#    Same specs as original (2GB+ RAM, 10GB+ disk)

# 2. Clone repository
git clone <repo_url> /opt/tfm-inventory
cd /opt/tfm-inventory

# 3. Copy backup from offsite storage
#    (Assuming you have offsite backup copy)
cp /path/to/offsite/backup-latest.tar.gz ./backups/

# 4. Restore
./scripts/restore.sh backups/backup-latest.tar.gz --validate

# 5. Update DNS/load balancer to new IP

# 6. Verify services
curl http://localhost:8000/health
curl http://localhost:3000
```

**RTO**: <30 minutes (depends on provisioning speed)  
**RPO**: 1 day  
**Estimated Cost**: New hardware provisioning

---

### Scenario 3: Data Center Failure

**Detection**:
- Entire data center unreachable
- Multiple systems down simultaneously
- Network infrastructure down

**Recovery Steps**:
```bash
# 1. Activate secondary site (if available)
#    or failover to cloud provider

# 2. Provision new infrastructure
#    Clone repository on new infrastructure

# 3. Restore latest backup
git clone <repo> /opt/tfm-inventory
cd /opt/tfm-inventory
./scripts/restore.sh /offsite/backup-latest.tar.gz --validate

# 4. Update DNS to new location
#    (Allow 5-15 min for DNS propagation)

# 5. Notify users
#    "Service restored; data loss = last 1 day"
```

**RTO**: 30-60 minutes (depends on secondary readiness)  
**RPO**: 1 day  
**Prevention**: Maintain offsite backup copy at minimum

---

### Scenario 4: Application Crash / Memory Leak

**Detection**:
- Backend crashes and won't restart
- Frontend crashes
- Memory continuously growing

**Recovery Steps**:
```bash
# Docker mode:
docker-compose logs backend | tail -100

# If memory leak:
docker-compose restart backend

# If crash persists:
git log --oneline | head -10
git revert <commit-hash>
./deploy.sh production

# Standalone mode:
tail -100 logs/backend.log

pkill -9 -f uvicorn
./start_server.sh

# If crash persists:
git revert <problematic-commit>
./start_server.sh
```

**RTO**: <5 minutes (restart)  
**RPO**: 0 (no data loss, running services)

---

### Scenario 5: Disk Full

**Detection**:
- `df -h` shows 100% usage
- Write operations failing
- Backup script failing

**Recovery Steps**:
```bash
# 1. Identify large directories
du -sh /* | sort -rh | head -10

# 2. Clean old backups
find backups/ -name "*.tar.gz" -mtime +7 -delete

# 3. Clear logs if very large
find logs/ -name "*.log" -mtime +30 -delete

# 4. Extend disk volume
#    (Depends on cloud provider or physical hardware)

# 5. Verify
df -h
```

**RTO**: <15 minutes  
**RPO**: 0 (no data loss)

---

### Scenario 6: Network Isolation / CORS Issues

**Detection**:
- Frontend can't reach backend API
- CORS errors in browser console
- API reachable locally but not from network

**Recovery Steps**:
```bash
# 1. Check network connectivity
ping <backend-ip>
curl -v http://backend-ip:8000/health

# 2. Check CORS configuration
docker-compose exec backend python -c "
from backend.config import ALLOWED_ORIGINS
print(ALLOWED_ORIGINS)
"

# 3. Update CORS if needed
# Edit inventory.env:
# EXTRA_ALLOWED_ORIGINS=<your-ip>

# 4. Restart backend
docker-compose restart backend

# 5. Verify
curl http://localhost:8000/health
```

**RTO**: <5 minutes  
**RPO**: 0

---

## Regular Testing

### Monthly Backup Test

Run on **staging environment** (not production):

```bash
# 1. List available backups
ls -lh backups/

# 2. Restore latest
./scripts/restore.sh backups/latest.tar.gz --validate

# 3. Verify checklist
- [ ] Restore completes without errors
- [ ] All services start correctly
- [ ] Database passes integrity check
- [ ] Item count matches expectation (e.g., 10K+ items)
- [ ] API responds at /health
- [ ] Frontend loads without errors
- [ ] Can login with test account
```

### Quarterly Full Failover Drill

Once per quarter, perform complete failover simulation:

```bash
# 1. Provision staging server with identical specs
# 2. Restore production backup
# 3. Run health checklist
# 4. Simulate 5 concurrent users (if load testing available)
# 5. Document any issues
# 6. Update this plan based on findings
```

### Annual Disaster Recovery Exercise

Once per year:
- Simulate data center failure
- Activate secondary site (if exists)
- Full restore on new infrastructure
- Involve all ops team members
- Document timeline and issues
- Update RTO/RPO estimates

---

## Prevention & Mitigation

| Layer | Prevention | Implementation |
|-------|-----------|-----------------|
| **Backup** | Daily automated | Cron jobs, 30-day rotation |
| **Offsite Backup** | Weekly copy to cloud | S3/GCS bucket, encrypted |
| **Monitoring** | Alert on issues | CPU >70%, disk >80%, API down |
| **Redundancy** | Secondary instance | v3 feature (not in v2 scope) |
| **Testing** | Monthly restore drill | Staging environment |
| **Documentation** | Up-to-date runbooks | Review quarterly |

---

## Offsite Backup Setup (Recommended)

To prevent total data loss in case of hardware failure:

```bash
# Weekly copy to cloud storage (add to cron)
0 4 * * 0 cd /opt/tfm-inventory && \
  gsutil -m cp backups/inventory-*.tar.gz \
  gs://your-backup-bucket/tfm-inventory/ || \
  aws s3 sync backups/ s3://your-bucket/tfm-inventory/

# Or to another server
0 4 * * 0 cd /opt/tfm-inventory && \
  rsync -avz backups/ backup-server:/backups/tfm-inventory/
```

---

## Communication Plan

### During Incident

1. **Immediate** (notify immediately):
   - CEO / Project Lead
   - Affected users
   - Operations team

2. **Message Template**:
   ```
   Service Status: [DEGRADED|DOWN]
   Impact: Inventory system unavailable
   ETA: <estimated recovery time>
   Action: We are restoring from backup
   ```

3. **Updates**: Every 5 minutes or when status changes

### After Recovery

1. **Post-incident Review**: Within 48 hours
   - What failed?
   - Why did it fail?
   - How do we prevent it?
   - Update this plan

2. **Root Cause Analysis**: Within 1 week
3. **Implement Fixes**: Within 2 weeks

---

## Success Criteria

For recovery to be considered successful:

- [ ] Restore completes in <10 minutes (target)
- [ ] Zero data loss (max 1 day RPO acceptable)
- [ ] All services healthy post-restore
- [ ] Users can login and access inventory
- [ ] API responds at /health with 200 OK
- [ ] Database integrity verified
- [ ] Audit logs preserved (immutable)
- [ ] Monthly test succeeds 100%

---

## Contacts & Escalation

| Role | Name | Contact | Hours |
|------|------|---------|-------|
| On-call Ops | [Name] | [Phone] | 24/7 |
| Database Admin | [Name] | [Email] | Business hours |
| Infrastructure | [Name] | [Email] | Business hours |
| CEO / Product | [Name] | [Phone] | Escalation only |

---

## Appendix: Recovery Time Estimates

| Scenario | Time | Notes |
|----------|------|-------|
| Restart service | 2-3 min | Quick fix for most issues |
| Restore from backup | 8-10 min | DB restore + service startup |
| New hardware | 20-30 min | Provisioning + restore |
| Data center failover | 30-60 min | Depends on secondary readiness |
| Network reconfiguration | 5-15 min | DNS + CORS setup |

---

**Version**: 1.0  
**Last Updated**: 2026-04-22  
**Last Tested**: [Date]  
**Owner**: Operations Team  
**Next Review**: 2026-05-22