feat(6): phase 6 plan 02 - operational runbook and documentation
- Created OPERATIONAL_RUNBOOK.md: comprehensive step-by-step procedures for both Docker and Standalone deployment modes covering deployment, daily ops, troubleshooting, backup/restore, disaster recovery, scaling, and updates - Created HEALTH_MONITORING_CHECKLIST.md: daily/weekly/monthly health check procedures with alert thresholds and quick troubleshooting reference - Created DISASTER_RECOVERY_PLAN.md: detailed procedures for 6 failure scenarios (database corruption, hardware failure, data center failure, app crash, disk full, network isolation) with RTO/RPO targets - Created CONFIGURATION_REFERENCE.md: complete documentation of all inventory.env parameters for both deployment modes with common scenarios and troubleshooting - Created EMERGENCY_PROCEDURES.md: quick-reference incident response playbook with 7 critical scenarios, decision tree, escalation path, and printable cheat sheet - Created scripts/backup.sh: automated backup script supporting both Docker and Standalone with integrity verification and retention management - Created scripts/restore.sh: restore script with triple confirmation, safety backups, and validation tests for both deployment modes - Created config/backup-cron.sh: installer for daily/weekly automated backup cron jobs (2 AM daily, 3 AM Sunday) All documentation covers dual-deployment modes with shared configuration files. Documentation is operator-ready with copy-paste commands and clear expected outputs.
This commit is contained in:
388
docs/DISASTER_RECOVERY_PLAN.md
Normal file
388
docs/DISASTER_RECOVERY_PLAN.md
Normal file
@@ -0,0 +1,388 @@
|
||||
# Disaster Recovery Plan
|
||||
|
||||
**Objective**: Restore production service within 10 minutes and zero data loss.
|
||||
**Status**: Active
|
||||
**Last Tested**: [Date]
|
||||
**Next Review**: 2026-05-22
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document outlines procedures for recovering from various failure scenarios. The system uses automated daily backups with a 1-day RPO (Recovery Point Objective) and aims for <10 minute RTO (Recovery Time Objective).
|
||||
|
||||
---
|
||||
|
||||
## Scenarios & Recovery Procedures
|
||||
|
||||
### Scenario 1: Database Corrupted
|
||||
|
||||
**Detection**:
|
||||
- Integrity check fails: `PRAGMA integrity_check;`
|
||||
- Data unexpectedly missing
|
||||
- Queries returning errors
|
||||
|
||||
**Recovery Steps (Docker)**:
|
||||
```bash
|
||||
# 1. Verify corruption
|
||||
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA integrity_check;"
|
||||
|
||||
# 2. Stop services
|
||||
docker-compose down
|
||||
|
||||
# 3. Restore from backup
|
||||
./scripts/restore.sh backups/latest.tar.gz --validate
|
||||
|
||||
# 4. Verify data integrity
|
||||
docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"
|
||||
```
|
||||
|
||||
**Recovery Steps (Standalone)**:
|
||||
```bash
|
||||
# 1. Verify corruption
|
||||
sqlite3 data/inventory.db "PRAGMA integrity_check;"
|
||||
|
||||
# 2. Stop services
|
||||
pkill -f uvicorn
|
||||
pkill -f "next start"
|
||||
|
||||
# 3. Restore from backup
|
||||
./scripts/restore.sh backups/latest.tar.gz
|
||||
|
||||
# 4. Start services
|
||||
./start_server.sh
|
||||
|
||||
# 5. Verify
|
||||
sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"
|
||||
```
|
||||
|
||||
**RTO**: <10 minutes
|
||||
**RPO**: 1 day
|
||||
**Notify Users**: If data loss within last 24 hours
|
||||
|
||||
---
|
||||
|
||||
### Scenario 2: Complete Hardware Failure
|
||||
|
||||
**Detection**:
|
||||
- Server doesn't boot
|
||||
- Server not reachable on network
|
||||
- Docker daemon won't start
|
||||
|
||||
**Recovery Steps**:
|
||||
```bash
|
||||
# 1. Provision new Ubuntu 22.04 LTS server
|
||||
# Same specs as original (2GB+ RAM, 10GB+ disk)
|
||||
|
||||
# 2. Clone repository
|
||||
git clone <repo_url> /opt/tfm-inventory
|
||||
cd /opt/tfm-inventory
|
||||
|
||||
# 3. Copy backup from offsite storage
|
||||
# (Assuming you have offsite backup copy)
|
||||
cp /path/to/offsite/backup-latest.tar.gz ./backups/
|
||||
|
||||
# 4. Restore
|
||||
./scripts/restore.sh backups/backup-latest.tar.gz --validate
|
||||
|
||||
# 5. Update DNS/load balancer to new IP
|
||||
|
||||
# 6. Verify services
|
||||
curl http://localhost:8000/health
|
||||
curl http://localhost:3000
|
||||
```
|
||||
|
||||
**RTO**: <30 minutes (depends on provisioning speed)
|
||||
**RPO**: 1 day
|
||||
**Estimated Cost**: New hardware provisioning
|
||||
|
||||
---
|
||||
|
||||
### Scenario 3: Data Center Failure
|
||||
|
||||
**Detection**:
|
||||
- Entire data center unreachable
|
||||
- Multiple systems down simultaneously
|
||||
- Network infrastructure down
|
||||
|
||||
**Recovery Steps**:
|
||||
```bash
|
||||
# 1. Activate secondary site (if available)
|
||||
# or failover to cloud provider
|
||||
|
||||
# 2. Provision new infrastructure
|
||||
# Clone repository on new infrastructure
|
||||
|
||||
# 3. Restore latest backup
|
||||
git clone <repo> /opt/tfm-inventory
|
||||
cd /opt/tfm-inventory
|
||||
./scripts/restore.sh /offsite/backup-latest.tar.gz --validate
|
||||
|
||||
# 4. Update DNS to new location
|
||||
# (Allow 5-15 min for DNS propagation)
|
||||
|
||||
# 5. Notify users
|
||||
# "Service restored; data loss = last 1 day"
|
||||
```
|
||||
|
||||
**RTO**: 30-60 minutes (depends on secondary readiness)
|
||||
**RPO**: 1 day
|
||||
**Prevention**: Maintain offsite backup copy at minimum
|
||||
|
||||
---
|
||||
|
||||
### Scenario 4: Application Crash / Memory Leak
|
||||
|
||||
**Detection**:
|
||||
- Backend crashes and won't restart
|
||||
- Frontend crashes
|
||||
- Memory continuously growing
|
||||
|
||||
**Recovery Steps**:
|
||||
```bash
|
||||
# Docker mode:
|
||||
docker-compose logs backend | tail -100
|
||||
|
||||
# If memory leak:
|
||||
docker-compose restart backend
|
||||
|
||||
# If crash persists:
|
||||
git log --oneline | head -10
|
||||
git revert <commit-hash>
|
||||
./deploy.sh production
|
||||
|
||||
# Standalone mode:
|
||||
tail -100 logs/backend.log
|
||||
|
||||
pkill -9 -f uvicorn
|
||||
./start_server.sh
|
||||
|
||||
# If crash persists:
|
||||
git revert <problematic-commit>
|
||||
./start_server.sh
|
||||
```
|
||||
|
||||
**RTO**: <5 minutes (restart)
|
||||
**RPO**: 0 (no data loss, running services)
|
||||
|
||||
---
|
||||
|
||||
### Scenario 5: Disk Full
|
||||
|
||||
**Detection**:
|
||||
- `df -h` shows 100% usage
|
||||
- Write operations failing
|
||||
- Backup script failing
|
||||
|
||||
**Recovery Steps**:
|
||||
```bash
|
||||
# 1. Identify large directories
|
||||
du -sh /* | sort -rh | head -10
|
||||
|
||||
# 2. Clean old backups
|
||||
find backups/ -name "*.tar.gz" -mtime +7 -delete
|
||||
|
||||
# 3. Clear logs if very large
|
||||
find logs/ -name "*.log" -mtime +30 -delete
|
||||
|
||||
# 4. Extend disk volume
|
||||
# (Depends on cloud provider or physical hardware)
|
||||
|
||||
# 5. Verify
|
||||
df -h
|
||||
```
|
||||
|
||||
**RTO**: <15 minutes
|
||||
**RPO**: 0 (no data loss)
|
||||
|
||||
---
|
||||
|
||||
### Scenario 6: Network Isolation / CORS Issues
|
||||
|
||||
**Detection**:
|
||||
- Frontend can't reach backend API
|
||||
- CORS errors in browser console
|
||||
- API reachable locally but not from network
|
||||
|
||||
**Recovery Steps**:
|
||||
```bash
|
||||
# 1. Check network connectivity
|
||||
ping <backend-ip>
|
||||
curl -v http://backend-ip:8000/health
|
||||
|
||||
# 2. Check CORS configuration
|
||||
docker-compose exec backend python -c "
|
||||
from backend.config import ALLOWED_ORIGINS
|
||||
print(ALLOWED_ORIGINS)
|
||||
"
|
||||
|
||||
# 3. Update CORS if needed
|
||||
# Edit inventory.env:
|
||||
# EXTRA_ALLOWED_ORIGINS=<your-ip>
|
||||
|
||||
# 4. Restart backend
|
||||
docker-compose restart backend
|
||||
|
||||
# 5. Verify
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
**RTO**: <5 minutes
|
||||
**RPO**: 0
|
||||
|
||||
---
|
||||
|
||||
## Regular Testing
|
||||
|
||||
### Monthly Backup Test
|
||||
|
||||
Run on **staging environment** (not production):
|
||||
|
||||
```bash
|
||||
# 1. List available backups
|
||||
ls -lh backups/
|
||||
|
||||
# 2. Restore latest
|
||||
./scripts/restore.sh backups/latest.tar.gz --validate
|
||||
|
||||
# 3. Verify checklist
|
||||
- [ ] Restore completes without errors
|
||||
- [ ] All services start correctly
|
||||
- [ ] Database passes integrity check
|
||||
- [ ] Item count matches expectation (e.g., 10K+ items)
|
||||
- [ ] API responds at /health
|
||||
- [ ] Frontend loads without errors
|
||||
- [ ] Can login with test account
|
||||
```
|
||||
|
||||
### Quarterly Full Failover Drill
|
||||
|
||||
Once per quarter, perform complete failover simulation:
|
||||
|
||||
```bash
|
||||
# 1. Provision staging server with identical specs
|
||||
# 2. Restore production backup
|
||||
# 3. Run health checklist
|
||||
# 4. Simulate 5 concurrent users (if load testing available)
|
||||
# 5. Document any issues
|
||||
# 6. Update this plan based on findings
|
||||
```
|
||||
|
||||
### Annual Disaster Recovery Exercise
|
||||
|
||||
Once per year:
|
||||
- Simulate data center failure
|
||||
- Activate secondary site (if exists)
|
||||
- Full restore on new infrastructure
|
||||
- Involve all ops team members
|
||||
- Document timeline and issues
|
||||
- Update RTO/RPO estimates
|
||||
|
||||
---
|
||||
|
||||
## Prevention & Mitigation
|
||||
|
||||
| Layer | Prevention | Implementation |
|
||||
|-------|-----------|-----------------|
|
||||
| **Backup** | Daily automated | Cron jobs, 30-day rotation |
|
||||
| **Offsite Backup** | Weekly copy to cloud | S3/GCS bucket, encrypted |
|
||||
| **Monitoring** | Alert on issues | CPU >70%, disk >80%, API down |
|
||||
| **Redundancy** | Secondary instance | v3 feature (not in v2 scope) |
|
||||
| **Testing** | Monthly restore drill | Staging environment |
|
||||
| **Documentation** | Up-to-date runbooks | Review quarterly |
|
||||
|
||||
---
|
||||
|
||||
## Offsite Backup Setup (Recommended)
|
||||
|
||||
To prevent total data loss in case of hardware failure:
|
||||
|
||||
```bash
|
||||
# Weekly copy to cloud storage (add to cron)
|
||||
0 4 * * 0 cd /opt/tfm-inventory && \
|
||||
gsutil -m cp backups/inventory-*.tar.gz \
|
||||
gs://your-backup-bucket/tfm-inventory/ || \
|
||||
aws s3 sync backups/ s3://your-bucket/tfm-inventory/
|
||||
|
||||
# Or to another server
|
||||
0 4 * * 0 cd /opt/tfm-inventory && \
|
||||
rsync -avz backups/ backup-server:/backups/tfm-inventory/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Communication Plan
|
||||
|
||||
### During Incident
|
||||
|
||||
1. **Immediate** (notify immediately):
|
||||
- CEO / Project Lead
|
||||
- Affected users
|
||||
- Operations team
|
||||
|
||||
2. **Message Template**:
|
||||
```
|
||||
Service Status: [DEGRADED|DOWN]
|
||||
Impact: Inventory system unavailable
|
||||
ETA: <estimated recovery time>
|
||||
Action: We are restoring from backup
|
||||
```
|
||||
|
||||
3. **Updates**: Every 5 minutes or when status changes
|
||||
|
||||
### After Recovery
|
||||
|
||||
1. **Post-incident Review**: Within 48 hours
|
||||
- What failed?
|
||||
- Why did it fail?
|
||||
- How do we prevent it?
|
||||
- Update this plan
|
||||
|
||||
2. **Root Cause Analysis**: Within 1 week
|
||||
3. **Implement Fixes**: Within 2 weeks
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
For recovery to be considered successful:
|
||||
|
||||
- [ ] Restore completes in <10 minutes (target)
|
||||
- [ ] Zero data loss (max 1 day RPO acceptable)
|
||||
- [ ] All services healthy post-restore
|
||||
- [ ] Users can login and access inventory
|
||||
- [ ] API responds at /health with 200 OK
|
||||
- [ ] Database integrity verified
|
||||
- [ ] Audit logs preserved (immutable)
|
||||
- [ ] Monthly test succeeds 100%
|
||||
|
||||
---
|
||||
|
||||
## Contacts & Escalation
|
||||
|
||||
| Role | Name | Contact | Hours |
|
||||
|------|------|---------|-------|
|
||||
| On-call Ops | [Name] | [Phone] | 24/7 |
|
||||
| Database Admin | [Name] | [Email] | Business hours |
|
||||
| Infrastructure | [Name] | [Email] | Business hours |
|
||||
| CEO / Product | [Name] | [Phone] | Escalation only |
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Recovery Time Estimates
|
||||
|
||||
| Scenario | Time | Notes |
|
||||
|----------|------|-------|
|
||||
| Restart service | 2-3 min | Quick fix for most issues |
|
||||
| Restore from backup | 8-10 min | DB restore + service startup |
|
||||
| New hardware | 20-30 min | Provisioning + restore |
|
||||
| Data center failover | 30-60 min | Depends on secondary readiness |
|
||||
| Network reconfiguration | 5-15 min | DNS + CORS setup |
|
||||
|
||||
---
|
||||
|
||||
**Version**: 1.0
|
||||
**Last Updated**: 2026-04-22
|
||||
**Last Tested**: [Date]
|
||||
**Owner**: Operations Team
|
||||
**Next Review**: 2026-05-22
|
||||
Reference in New Issue
Block a user