Daniel Bedeleanu
fc149184e9
- Created OPERATIONAL_RUNBOOK.md: comprehensive step-by-step procedures for both Docker and Standalone deployment modes covering deployment, daily ops, troubleshooting, backup/restore, disaster recovery, scaling, and updates - Created HEALTH_MONITORING_CHECKLIST.md: daily/weekly/monthly health check procedures with alert thresholds and quick troubleshooting reference - Created DISASTER_RECOVERY_PLAN.md: detailed procedures for 6 failure scenarios (database corruption, hardware failure, data center failure, app crash, disk full, network isolation) with RTO/RPO targets - Created CONFIGURATION_REFERENCE.md: complete documentation of all inventory.env parameters for both deployment modes with common scenarios and troubleshooting - Created EMERGENCY_PROCEDURES.md: quick-reference incident response playbook with 7 critical scenarios, decision tree, escalation path, and printable cheat sheet - Created scripts/backup.sh: automated backup script supporting both Docker and Standalone with integrity verification and retention management - Created scripts/restore.sh: restore script with triple confirmation, safety backups, and validation tests for both deployment modes - Created config/backup-cron.sh: installer for daily/weekly automated backup cron jobs (2 AM daily, 3 AM Sunday) All documentation covers dual-deployment modes with shared configuration files. Documentation is operator-ready with copy-paste commands and clear expected outputs.
389 lines
8.6 KiB
Markdown
389 lines
8.6 KiB
Markdown
# Disaster Recovery Plan
|
|
|
|
**Objective**: Restore production service within 10 minutes and zero data loss.
|
|
**Status**: Active
|
|
**Last Tested**: [Date]
|
|
**Next Review**: 2026-05-22
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
This document outlines procedures for recovering from various failure scenarios. The system uses automated daily backups with a 1-day RPO (Recovery Point Objective) and aims for <10 minute RTO (Recovery Time Objective).
|
|
|
|
---
|
|
|
|
## Scenarios & Recovery Procedures
|
|
|
|
### Scenario 1: Database Corrupted
|
|
|
|
**Detection**:
|
|
- Integrity check fails: `PRAGMA integrity_check;`
|
|
- Data unexpectedly missing
|
|
- Queries returning errors
|
|
|
|
**Recovery Steps (Docker)**:
|
|
```bash
|
|
# 1. Verify corruption
|
|
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA integrity_check;"
|
|
|
|
# 2. Stop services
|
|
docker-compose down
|
|
|
|
# 3. Restore from backup
|
|
./scripts/restore.sh backups/latest.tar.gz --validate
|
|
|
|
# 4. Verify data integrity
|
|
docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"
|
|
```
|
|
|
|
**Recovery Steps (Standalone)**:
|
|
```bash
|
|
# 1. Verify corruption
|
|
sqlite3 data/inventory.db "PRAGMA integrity_check;"
|
|
|
|
# 2. Stop services
|
|
pkill -f uvicorn
|
|
pkill -f "next start"
|
|
|
|
# 3. Restore from backup
|
|
./scripts/restore.sh backups/latest.tar.gz
|
|
|
|
# 4. Start services
|
|
./start_server.sh
|
|
|
|
# 5. Verify
|
|
sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"
|
|
```
|
|
|
|
**RTO**: <10 minutes
|
|
**RPO**: 1 day
|
|
**Notify Users**: If data loss within last 24 hours
|
|
|
|
---
|
|
|
|
### Scenario 2: Complete Hardware Failure
|
|
|
|
**Detection**:
|
|
- Server doesn't boot
|
|
- Server not reachable on network
|
|
- Docker daemon won't start
|
|
|
|
**Recovery Steps**:
|
|
```bash
|
|
# 1. Provision new Ubuntu 22.04 LTS server
|
|
# Same specs as original (2GB+ RAM, 10GB+ disk)
|
|
|
|
# 2. Clone repository
|
|
git clone <repo_url> /opt/tfm-inventory
|
|
cd /opt/tfm-inventory
|
|
|
|
# 3. Copy backup from offsite storage
|
|
# (Assuming you have offsite backup copy)
|
|
cp /path/to/offsite/backup-latest.tar.gz ./backups/
|
|
|
|
# 4. Restore
|
|
./scripts/restore.sh backups/backup-latest.tar.gz --validate
|
|
|
|
# 5. Update DNS/load balancer to new IP
|
|
|
|
# 6. Verify services
|
|
curl http://localhost:8000/health
|
|
curl http://localhost:3000
|
|
```
|
|
|
|
**RTO**: <30 minutes (depends on provisioning speed)
|
|
**RPO**: 1 day
|
|
**Estimated Cost**: New hardware provisioning
|
|
|
|
---
|
|
|
|
### Scenario 3: Data Center Failure
|
|
|
|
**Detection**:
|
|
- Entire data center unreachable
|
|
- Multiple systems down simultaneously
|
|
- Network infrastructure down
|
|
|
|
**Recovery Steps**:
|
|
```bash
|
|
# 1. Activate secondary site (if available)
|
|
# or failover to cloud provider
|
|
|
|
# 2. Provision new infrastructure
|
|
# Clone repository on new infrastructure
|
|
|
|
# 3. Restore latest backup
|
|
git clone <repo> /opt/tfm-inventory
|
|
cd /opt/tfm-inventory
|
|
./scripts/restore.sh /offsite/backup-latest.tar.gz --validate
|
|
|
|
# 4. Update DNS to new location
|
|
# (Allow 5-15 min for DNS propagation)
|
|
|
|
# 5. Notify users
|
|
# "Service restored; data loss = last 1 day"
|
|
```
|
|
|
|
**RTO**: 30-60 minutes (depends on secondary readiness)
|
|
**RPO**: 1 day
|
|
**Prevention**: Maintain offsite backup copy at minimum
|
|
|
|
---
|
|
|
|
### Scenario 4: Application Crash / Memory Leak
|
|
|
|
**Detection**:
|
|
- Backend crashes and won't restart
|
|
- Frontend crashes
|
|
- Memory continuously growing
|
|
|
|
**Recovery Steps**:
|
|
```bash
|
|
# Docker mode:
|
|
docker-compose logs backend | tail -100
|
|
|
|
# If memory leak:
|
|
docker-compose restart backend
|
|
|
|
# If crash persists:
|
|
git log --oneline | head -10
|
|
git revert <commit-hash>
|
|
./deploy.sh production
|
|
|
|
# Standalone mode:
|
|
tail -100 logs/backend.log
|
|
|
|
pkill -9 -f uvicorn
|
|
./start_server.sh
|
|
|
|
# If crash persists:
|
|
git revert <problematic-commit>
|
|
./start_server.sh
|
|
```
|
|
|
|
**RTO**: <5 minutes (restart)
|
|
**RPO**: 0 (no data loss, running services)
|
|
|
|
---
|
|
|
|
### Scenario 5: Disk Full
|
|
|
|
**Detection**:
|
|
- `df -h` shows 100% usage
|
|
- Write operations failing
|
|
- Backup script failing
|
|
|
|
**Recovery Steps**:
|
|
```bash
|
|
# 1. Identify large directories
|
|
du -sh /* | sort -rh | head -10
|
|
|
|
# 2. Clean old backups
|
|
find backups/ -name "*.tar.gz" -mtime +7 -delete
|
|
|
|
# 3. Clear logs if very large
|
|
find logs/ -name "*.log" -mtime +30 -delete
|
|
|
|
# 4. Extend disk volume
|
|
# (Depends on cloud provider or physical hardware)
|
|
|
|
# 5. Verify
|
|
df -h
|
|
```
|
|
|
|
**RTO**: <15 minutes
|
|
**RPO**: 0 (no data loss)
|
|
|
|
---
|
|
|
|
### Scenario 6: Network Isolation / CORS Issues
|
|
|
|
**Detection**:
|
|
- Frontend can't reach backend API
|
|
- CORS errors in browser console
|
|
- API reachable locally but not from network
|
|
|
|
**Recovery Steps**:
|
|
```bash
|
|
# 1. Check network connectivity
|
|
ping <backend-ip>
|
|
curl -v http://backend-ip:8000/health
|
|
|
|
# 2. Check CORS configuration
|
|
docker-compose exec backend python -c "
|
|
from backend.config import ALLOWED_ORIGINS
|
|
print(ALLOWED_ORIGINS)
|
|
"
|
|
|
|
# 3. Update CORS if needed
|
|
# Edit inventory.env:
|
|
# EXTRA_ALLOWED_ORIGINS=<your-ip>
|
|
|
|
# 4. Restart backend
|
|
docker-compose restart backend
|
|
|
|
# 5. Verify
|
|
curl http://localhost:8000/health
|
|
```
|
|
|
|
**RTO**: <5 minutes
|
|
**RPO**: 0
|
|
|
|
---
|
|
|
|
## Regular Testing
|
|
|
|
### Monthly Backup Test
|
|
|
|
Run on **staging environment** (not production):
|
|
|
|
```bash
|
|
# 1. List available backups
|
|
ls -lh backups/
|
|
|
|
# 2. Restore latest
|
|
./scripts/restore.sh backups/latest.tar.gz --validate
|
|
|
|
# 3. Verify checklist
|
|
- [ ] Restore completes without errors
|
|
- [ ] All services start correctly
|
|
- [ ] Database passes integrity check
|
|
- [ ] Item count matches expectation (e.g., 10K+ items)
|
|
- [ ] API responds at /health
|
|
- [ ] Frontend loads without errors
|
|
- [ ] Can login with test account
|
|
```
|
|
|
|
### Quarterly Full Failover Drill
|
|
|
|
Once per quarter, perform complete failover simulation:
|
|
|
|
```bash
|
|
# 1. Provision staging server with identical specs
|
|
# 2. Restore production backup
|
|
# 3. Run health checklist
|
|
# 4. Simulate 5 concurrent users (if load testing available)
|
|
# 5. Document any issues
|
|
# 6. Update this plan based on findings
|
|
```
|
|
|
|
### Annual Disaster Recovery Exercise
|
|
|
|
Once per year:
|
|
- Simulate data center failure
|
|
- Activate secondary site (if exists)
|
|
- Full restore on new infrastructure
|
|
- Involve all ops team members
|
|
- Document timeline and issues
|
|
- Update RTO/RPO estimates
|
|
|
|
---
|
|
|
|
## Prevention & Mitigation
|
|
|
|
| Layer | Prevention | Implementation |
|
|
|-------|-----------|-----------------|
|
|
| **Backup** | Daily automated | Cron jobs, 30-day rotation |
|
|
| **Offsite Backup** | Weekly copy to cloud | S3/GCS bucket, encrypted |
|
|
| **Monitoring** | Alert on issues | CPU >70%, disk >80%, API down |
|
|
| **Redundancy** | Secondary instance | v3 feature (not in v2 scope) |
|
|
| **Testing** | Monthly restore drill | Staging environment |
|
|
| **Documentation** | Up-to-date runbooks | Review quarterly |
|
|
|
|
---
|
|
|
|
## Offsite Backup Setup (Recommended)
|
|
|
|
To prevent total data loss in case of hardware failure:
|
|
|
|
```bash
|
|
# Weekly copy to cloud storage (add to cron)
|
|
0 4 * * 0 cd /opt/tfm-inventory && \
|
|
gsutil -m cp backups/inventory-*.tar.gz \
|
|
gs://your-backup-bucket/tfm-inventory/ || \
|
|
aws s3 sync backups/ s3://your-bucket/tfm-inventory/
|
|
|
|
# Or to another server
|
|
0 4 * * 0 cd /opt/tfm-inventory && \
|
|
rsync -avz backups/ backup-server:/backups/tfm-inventory/
|
|
```
|
|
|
|
---
|
|
|
|
## Communication Plan
|
|
|
|
### During Incident
|
|
|
|
1. **Immediate** (notify immediately):
|
|
- CEO / Project Lead
|
|
- Affected users
|
|
- Operations team
|
|
|
|
2. **Message Template**:
|
|
```
|
|
Service Status: [DEGRADED|DOWN]
|
|
Impact: Inventory system unavailable
|
|
ETA: <estimated recovery time>
|
|
Action: We are restoring from backup
|
|
```
|
|
|
|
3. **Updates**: Every 5 minutes or when status changes
|
|
|
|
### After Recovery
|
|
|
|
1. **Post-incident Review**: Within 48 hours
|
|
- What failed?
|
|
- Why did it fail?
|
|
- How do we prevent it?
|
|
- Update this plan
|
|
|
|
2. **Root Cause Analysis**: Within 1 week
|
|
3. **Implement Fixes**: Within 2 weeks
|
|
|
|
---
|
|
|
|
## Success Criteria
|
|
|
|
For recovery to be considered successful:
|
|
|
|
- [ ] Restore completes in <10 minutes (target)
|
|
- [ ] Zero data loss (max 1 day RPO acceptable)
|
|
- [ ] All services healthy post-restore
|
|
- [ ] Users can login and access inventory
|
|
- [ ] API responds at /health with 200 OK
|
|
- [ ] Database integrity verified
|
|
- [ ] Audit logs preserved (immutable)
|
|
- [ ] Monthly test succeeds 100%
|
|
|
|
---
|
|
|
|
## Contacts & Escalation
|
|
|
|
| Role | Name | Contact | Hours |
|
|
|------|------|---------|-------|
|
|
| On-call Ops | [Name] | [Phone] | 24/7 |
|
|
| Database Admin | [Name] | [Email] | Business hours |
|
|
| Infrastructure | [Name] | [Email] | Business hours |
|
|
| CEO / Product | [Name] | [Phone] | Escalation only |
|
|
|
|
---
|
|
|
|
## Appendix: Recovery Time Estimates
|
|
|
|
| Scenario | Time | Notes |
|
|
|----------|------|-------|
|
|
| Restart service | 2-3 min | Quick fix for most issues |
|
|
| Restore from backup | 8-10 min | DB restore + service startup |
|
|
| New hardware | 20-30 min | Provisioning + restore |
|
|
| Data center failover | 30-60 min | Depends on secondary readiness |
|
|
| Network reconfiguration | 5-15 min | DNS + CORS setup |
|
|
|
|
---
|
|
|
|
**Version**: 1.0
|
|
**Last Updated**: 2026-04-22
|
|
**Last Tested**: [Date]
|
|
**Owner**: Operations Team
|
|
**Next Review**: 2026-05-22
|