# Phase 6, Plan 3: Backup/Restore & Operational Runbook

---

**plan**: 06-deployment-scale/03-backup-runbook  
**feature**: Automated backup/restore, disaster recovery, operational runbook  
**status**: Ready for execution  
**estimated_tasks**: 7  
**total_lines**: ~500 (backup script ~120, restore script ~100, runbook ~150, monitoring ~50, troubleshooting ~80)

---

## Overview

This plan ensures operational resilience through automated backup/restore and comprehensive documentation. It creates:

1. **Backup automation** — Daily/weekly scheduled backups of DB, config, certificates
2. **Restore validation** — Automated restore testing, zero-data-loss confirmation
3. **Operational runbook** — Step-by-step procedures for deployment, scaling, troubleshooting
4. **Disaster recovery** — RTO <10 minutes, RPO 1 day (configurable)
5. **Health monitoring** — Checklist-style monitoring for ops teams

**Success**: Backup/restore cycle tested and validated; runbook enables new ops to deploy and maintain without support.

---

## Tasks

### Task 1: Backup Automation Script
**File**: `scripts/backup.sh` (new, ~120 lines)  
**Status**: Ready  
**Description**: Automated backup of database, config, and certificates

**Content** (~120 lines):
```bash
#!/bin/bash
set -euo pipefail

# Phase 6, Plan 3, Task 1: Automated Backup Script
# Usage: ./backup.sh [daily|weekly|manual] [retention_days]

BACKUP_TYPE="${1:-manual}"
RETENTION_DAYS="${2:-30}"
BACKUP_DIR="./backups"
DATA_DIR="./data"
TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S)
BACKUP_FILE="$BACKUP_DIR/inventory-$TIMESTAMP.tar.gz"

# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; exit 1; }

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Verify Docker is running
docker ps > /dev/null 2>&1 || log_error "Docker daemon not running"

# 1. Verify services are running
log_info "Checking services..."
if ! docker-compose ps | grep -q "Up"; then
  log_warn "Not all services running; attempting to start..."
  docker-compose up -d
fi

# 2. Create backup tarball
log_info "Creating backup: $BACKUP_FILE"

# Stop backend to ensure DB consistency
log_info "Stopping backend service (DB consistency)..."
docker-compose stop backend

# Wait for graceful shutdown
sleep 2

# Create tarball
tar --exclude='$DATA_DIR/caddy_*' \
    --exclude='$BACKUP_DIR' \
    --exclude='.git' \
    --exclude='node_modules' \
    --exclude='.next' \
    -czf "$BACKUP_FILE" \
    "$DATA_DIR/inventory.db" \
    "$DATA_DIR/inventory.db-wal" \
    "$DATA_DIR/inventory.db-shm" \
    "config/" \
    "inventory.env" \
    2>/dev/null || log_error "Backup creation failed"

# Restart backend
log_info "Restarting backend service..."
docker-compose start backend

# Wait for backend to be ready
sleep 5

# Verify backend is healthy
if curl -sf "http://localhost:8000/health" > /dev/null; then
  log_info "Backend restored and healthy"
else
  log_warn "Backend not responding yet; check logs with 'docker-compose logs -f backend'"
fi

# 3. Verify backup integrity
log_info "Verifying backup integrity..."
if tar -tzf "$BACKUP_FILE" > /dev/null 2>&1; then
  log_info "✓ Backup verified"
else
  log_error "Backup corrupted"
fi

# 4. Log backup metadata
BACKUP_SIZE=$(du -h "$BACKUP_FILE" | cut -f1)
BACKUP_VERSION=$(cat VERSION.json 2>/dev/null | grep version | cut -d'"' -f4 || echo "unknown")
log_info "Backup size: $BACKUP_SIZE, Version: $BACKUP_VERSION"

# 5. Cleanup old backups
log_info "Cleaning up backups older than $RETENTION_DAYS days..."
find "$BACKUP_DIR" -name "inventory-*.tar.gz" -type f -mtime +$RETENTION_DAYS -delete

# 6. Create backup manifest
cat > "$BACKUP_DIR/MANIFEST.txt" << EOF
Backup Metadata
===============
Timestamp: $TIMESTAMP
Type: $BACKUP_TYPE
Retention: $RETENTION_DAYS days
Size: $BACKUP_SIZE
Version: $BACKUP_VERSION
Contents: DB, config, certificates (excluding certs)
Restored: Not yet

Command to restore:
./restore.sh $BACKUP_FILE
EOF

log_info "Backup complete: $BACKUP_FILE"
log_info "Retention policy: Delete after $RETENTION_DAYS days"
log_info "Next backup: $(date -d '+1 day' '+%Y-%m-%d')"
```

**Acceptance Criteria**:
- [ ] Creates gzip tarball of DB, config, certificates
- [ ] Stops backend before backup for consistency
- [ ] Restarts backend after backup
- [ ] Verifies tarball integrity
- [ ] Cleans up old backups based on retention
- [ ] Creates manifest with metadata
- [ ] Handles errors gracefully

**Testing**:
```bash
chmod +x scripts/backup.sh
./deploy.sh production
./scripts/backup.sh manual
# Verify backup created
ls -lh backups/
# Verify tarball integrity
tar -tzf backups/inventory-*.tar.gz | head
```

---

### Task 2: Restore & Validation Script
**File**: `scripts/restore.sh` (new, ~100 lines)  
**Status**: Ready  
**Description**: Restore from backup, validate data integrity

**Content** (~100 lines):
```bash
#!/bin/bash
set -euo pipefail

# Phase 6, Plan 3, Task 2: Restore from Backup
# Usage: ./restore.sh <backup_file> [--validate]

BACKUP_FILE="${1:-}"
VALIDATE="${2:---validate}"

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; exit 1; }

# Validate input
[[ -z "$BACKUP_FILE" ]] && log_error "Usage: ./restore.sh <backup_file>"
[[ ! -f "$BACKUP_FILE" ]] && log_error "Backup file not found: $BACKUP_FILE"

log_info "Restoring from: $BACKUP_FILE"

# Triple confirmation (security requirement)
echo "⚠️  WARNING: This will overwrite current data!"
read -p "Type 'RESTORE' to confirm (1/3): " confirm1
[[ "$confirm1" != "RESTORE" ]] && log_error "Restore cancelled"

read -p "Type 'RESTORE' again to confirm (2/3): " confirm2
[[ "$confirm2" != "RESTORE" ]] && log_error "Restore cancelled"

read -p "Type 'RESTORE' one more time to confirm (3/3): " confirm3
[[ "$confirm3" != "RESTORE" ]] && log_error "Restore cancelled"

log_warn "Proceeding with restore..."

# 1. Stop services
log_info "Stopping services..."
docker-compose down

# 2. Backup current data (safety copy)
log_info "Creating safety backup of current data..."
mkdir -p data/backups_before_restore
tar -czf "data/backups_before_restore/backup-before-restore-$(date +%s).tar.gz" \
    data/inventory.db data/inventory.db-* 2>/dev/null || true

# 3. Extract backup
log_info "Extracting backup..."
tar -xzf "$BACKUP_FILE" -C . || log_error "Backup extraction failed"

# 4. Verify essential files
log_info "Verifying restored files..."
[[ -f "data/inventory.db" ]] || log_error "Database not found in backup"
[[ -f "inventory.env" ]] || log_error "inventory.env not found in backup"

# 5. Restart services
log_info "Restarting services..."
docker-compose up -d

# 6. Wait for health
log_info "Waiting for services to be healthy..."
max_attempts=30
attempt=0
while [[ $attempt -lt $max_attempts ]]; do
  if curl -sf "http://localhost:8000/health" > /dev/null 2>&1; then
    log_info "Services healthy!"
    break
  fi
  attempt=$((attempt + 1))
  sleep 2
done

if [[ $attempt -eq $max_attempts ]]; then
  log_warn "Services did not become healthy within 60 seconds"
  docker-compose logs backend
  exit 1
fi

# 7. Validation tests (if requested)
if [[ "$VALIDATE" == "--validate" ]]; then
  log_info "Running validation tests..."
  
  # Test 1: Database accessible
  DB_ITEMS=$(docker-compose exec backend sqlite3 /app/data/inventory.db \
    "SELECT COUNT(*) FROM items" 2>/dev/null || echo "0")
  log_info "✓ Database has $DB_ITEMS items"
  
  # Test 2: API responsive
  if curl -sf "http://localhost:8000/health" > /dev/null; then
    log_info "✓ API health check passed"
  else
    log_error "API health check failed"
  fi
  
  # Test 3: Frontend loads
  if curl -sf "http://localhost:3000" > /dev/null 2>&1; then
    log_info "✓ Frontend loads"
  else
    log_warn "⚠ Frontend check failed (normal on first startup)"
  fi
fi

log_info "Restore complete!"
log_warn "Remember to verify data in the application before returning to production"
```

**Acceptance Criteria**:
- [ ] Prompts for triple confirmation
- [ ] Creates safety backup before restore
- [ ] Extracts tarball with proper permissions
- [ ] Validates essential files present
- [ ] Restarts all services
- [ ] Optional validation tests (DB count, API health, frontend)
- [ ] Clear error messages on failure

**Testing**:
```bash
chmod +x scripts/restore.sh
# Create a backup first
./scripts/backup.sh
# Simulate data corruption
rm data/inventory.db
# Restore and validate
./scripts/restore.sh backups/inventory-*.tar.gz --validate
# Verify data returned
```

---

### Task 3: Cron Job Configuration
**File**: `config/backup-cron.sh` (new, ~50 lines)  
**Status**: Ready  
**Description**: Setup automated daily/weekly backups via cron

**Content** (~50 lines):
```bash
#!/bin/bash
# Phase 6, Plan 3, Task 3: Install Cron Jobs
# Run with: sudo bash config/backup-cron.sh

DEPLOY_DIR=$(pwd)
CRON_SCHEDULE_DAILY="0 2 * * *"  # 2 AM every day
CRON_SCHEDULE_WEEKLY="0 3 * * 0" # 3 AM every Sunday

# Check if running with sudo
if [[ $EUID -ne 0 ]]; then
  echo "This script must be run with sudo"
  exit 1
fi

echo "Installing cron jobs for automated backups..."

# Install daily backup
(crontab -l 2>/dev/null | grep -v "inventory backup"; \
  echo "$CRON_SCHEDULE_DAILY cd $DEPLOY_DIR && bash scripts/backup.sh daily >> logs/backup-daily.log 2>&1") | \
  crontab -

# Install weekly backup
(crontab -l 2>/dev/null | grep -v "inventory backup"; \
  echo "$CRON_SCHEDULE_WEEKLY cd $DEPLOY_DIR && bash scripts/backup.sh weekly 90 >> logs/backup-weekly.log 2>&1") | \
  crontab -

echo "✓ Cron jobs installed"
echo "  Daily backup: $CRON_SCHEDULE_DAILY (retention: 30 days)"
echo "  Weekly backup: $CRON_SCHEDULE_WEEKLY (retention: 90 days)"
echo ""
echo "View active cron jobs:"
crontab -l | grep backup
```

**Acceptance Criteria**:
- [ ] Installs daily and weekly cron jobs
- [ ] Logs to files for audit trail
- [ ] Daily retention 30 days, weekly 90 days
- [ ] Can be installed/uninstalled without manual edits
- [ ] Works on Ubuntu 22.04+

**Testing**:
```bash
sudo bash config/backup-cron.sh
# Verify installation
sudo crontab -l | grep backup
# Simulate a run
cd /path/to/tfm-inventory && bash scripts/backup.sh daily
```

---

### Task 4: Operational Runbook
**File**: `docs/OPERATIONAL_RUNBOOK.md` (new, ~200 lines)  
**Status**: Ready  
**Description**: Step-by-step procedures for ops teams

**Content** (~200 lines):
```markdown
# Operational Runbook

**Audience**: Systems operators, site managers, DevOps teams  
**Target**: Minimal training required; step-by-step procedures

---

## 1. Initial Deployment

### Requirements
- Ubuntu 22.04 LTS or similar
- Docker, Docker Compose installed
- 2GB RAM, 10GB disk (recommended: 4GB/50GB for production)
- Internet access (first-time setup only)

### Steps

1. **Clone repository**
   ```bash
   git clone <repo_url> /opt/tfm-inventory
   cd /opt/tfm-inventory
   ```

2. **Configure environment**
   ```bash
   cp inventory.env.template inventory.env
   # Edit inventory.env with your settings:
   # - BACKEND_PORT, FRONTEND_PORT
   # - JWT_SECRET_KEY (generate: openssl rand -hex 32)
   # - AI settings (Gemini/Claude API keys)
   # - LDAP settings (if using enterprise auth)
   ```

3. **Deploy**
   ```bash
   chmod +x deploy.sh
   ./deploy.sh production
   ```

4. **Verify**
   - Frontend: http://your-server:3000
   - Backend API: http://your-server:8000
   - API Docs: http://your-server:8000/docs

5. **Create admin user**
   ```bash
   docker-compose exec backend python -c "
   from backend.db import SessionLocal, User
   db = SessionLocal()
   user = User(username='admin', hashed_password='...', is_admin=True)
   db.add(user)
   db.commit()
   "
   ```

---

## 2. Daily Operations

### Health Checks (Daily)

```bash
# Check all services
docker-compose ps
# Expected: All services "Up"

# Check API health
curl http://localhost:8000/health

# Check database size
du -h data/inventory.db

# Check logs for errors
docker-compose logs | grep ERROR
```

### Backup (Automated)

```bash
# Verify automatic backup ran
ls -lh backups/ | head -1

# Manual backup (if needed)
./scripts/backup.sh manual

# View backup schedule
sudo crontab -l | grep backup
```

### Monitoring

```bash
# Real-time logs
docker-compose logs -f

# Backend performance
docker stats --no-stream | grep backend

# Database status
docker-compose exec backend sqlite3 /app/data/inventory.db \
  "SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"
```

---

## 3. Troubleshooting

### Service won't start

```bash
# Check Docker daemon
docker ps

# Check port conflicts
netstat -tuln | grep 8000

# View service logs
docker-compose logs backend
docker-compose logs frontend
docker-compose logs proxy
```

### High CPU/Memory

```bash
# Identify container
docker stats

# Restart container
docker-compose restart backend

# Check for slow queries
docker-compose logs backend | grep "slow query"
```

### Database locked

```bash
# Restart backend
docker-compose restart backend

# Check WAL mode status
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA journal_mode;"
```

### HTTPS Certificate issues

```bash
# Certificates regenerated automatically
# If issues persist:
rm -rf data/caddy_*
docker-compose restart proxy
# Wait 30 seconds for new certs
```

---

## 4. Scaling Operations

### Adding Users (5+ concurrent)

Currently configured and tested for 5 concurrent users safely.

To support more users:
1. Increase backend memory: Edit docker-compose.yml
   ```yaml
   backend:
     mem_limit: 4g
   ```

2. Increase database connections:
   ```bash
   docker-compose exec backend \
     python -c "import backend.config; print(backend.config.DB_POOL_SIZE)"
   ```

3. Add read replicas (if needed, v3 feature)

### Database Growth (10K+ items)

As database grows beyond 10K items:
1. Monitor query performance: `PRAGMA optimize;`
2. Create indexes on frequently searched columns
3. Vacuum database: `VACUUM;`

---

## 5. Backup & Restore

### Automated Backups

```bash
# Cron jobs run automatically
# Daily: 2 AM, retention 30 days
# Weekly: 3 AM Sundays, retention 90 days

# Verify cron installation
sudo bash config/backup-cron.sh

# View backup history
ls -lh backups/
```

### Manual Restore

```bash
# List available backups
ls backups/

# Restore specific backup
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz

# Validate data after restore
curl http://localhost:8000/health
```

**RTO (Recovery Time Objective)**: <10 minutes  
**RPO (Recovery Point Objective)**: 1 day (daily backup)

---

## 6. Disaster Recovery

### Complete System Failure

1. **Restore on new server**
   ```bash
   # Fresh Ubuntu 22.04
   sudo apt-get update && sudo apt-get install -y docker.io docker-compose
   git clone <repo> /opt/tfm-inventory
   cd /opt/tfm-inventory
   ./scripts/restore.sh backups/latest.tar.gz --validate
   ```

2. **Verify data integrity**
   ```bash
   curl http://localhost:8000/health
   # Check item count in database
   ```

3. **Return to production**
   - Update DNS/load balancer
   - Notify users

### Data Corruption

1. **Investigate**
   ```bash
   docker-compose exec backend python -c "
   import sqlite3
   db = sqlite3.connect('/app/data/inventory.db')
   # Run integrity check
   print(db.execute('PRAGMA integrity_check').fetchall())
   "
   ```

2. **If corrupted**
   - Restore from last backup: `./scripts/restore.sh backups/latest.tar.gz`
   - Notify affected users of recovery

---

## 7. Updates & Upgrades

### Patch Update (v1.14.x → v1.14.y)

```bash
# Backup first
./scripts/backup.sh manual

# Pull latest code
git pull origin main

# Rebuild and restart
./deploy.sh production --rebuild

# Verify
curl http://localhost:8000/health
```

### Major Update (v1.x → v2.x)

```bash
# Create backup before proceeding
./scripts/backup.sh manual

# Review CHANGELOG for breaking changes
cat CHANGELOG.md | grep "v2.0"

# Follow upgrade guide
cat docs/UPGRADE_GUIDE.md

# Test in staging first
./scripts/restore.sh backups/production.tar.gz
# (Test on staging environment)

# Proceed to production
git checkout v2.0
./deploy.sh production --rebuild
```

---

## 8. Emergency Contacts

- Developer Support: dev@example.com
- Infrastructure: ops@example.com
- 24/7 On-call: [contact info]

---

**Last Updated**: 2026-04-22  
**Version**: 1.0  
**Maintainer**: Operations Team
```

**Acceptance Criteria**:
- [ ] Covers full deployment lifecycle
- [ ] Health check procedures documented
- [ ] Troubleshooting section covers common issues
- [ ] Scaling guidance clear
- [ ] Backup/restore procedures step-by-step
- [ ] Written for non-technical audience
- [ ] Emergency contacts and escalation paths

---

### Task 5: Health Monitoring Checklist
**File**: `docs/HEALTH_MONITORING_CHECKLIST.md` (new, ~80 lines)  
**Status**: Ready  
**Description**: Daily/weekly health checks for ops teams

**Content** (~80 lines):
```markdown
# Health Monitoring Checklist

Use this checklist for daily/weekly health reviews.

## Daily (5 minutes)

- [ ] All services running: `docker-compose ps`
  - Expected: backend, frontend, proxy all "Up"
- [ ] API responsive: `curl http://localhost:8000/health`
  - Expected: 200 OK, response <100ms
- [ ] Frontend loads: `curl http://localhost:3000/`
  - Expected: 200 OK
- [ ] Recent errors in logs: `docker-compose logs | grep ERROR | tail -5`
  - Action: Investigate any ERROR-level logs
- [ ] Database accessible: `docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"`
  - Action: If fails, restart backend

## Weekly (15 minutes)

- [ ] Backup completed: `ls -lh backups/ | head -1`
  - Check timestamp is within last 24 hours
- [ ] Disk usage: `du -sh data/ config/ backups/`
  - Expected: data/ <5GB, backups/ <10GB (5 weeks @ 2GB/week)
  - Action: If backups >10GB, verify cron retention is set correctly
- [ ] Database size: `docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT page_count * page_size / (1024*1024) FROM pragma_page_count(), pragma_page_size();"`
  - Action: If >1GB, consider optimization
- [ ] Service resource usage: `docker stats --no-stream`
  - Expected: backend <70% CPU, <500MB RAM
  - Action: If exceeds, investigate slow queries
- [ ] Restore test: `./scripts/backup.sh manual`
  - Action: Run monthly
- [ ] Update check: `git status`
  - Action: Review available updates

## Monthly (30 minutes)

- [ ] Restore from backup test
  ```bash
  # On staging environment
  ./scripts/restore.sh backups/latest.tar.gz --validate
  ```
  - Action: Confirm zero data loss, all services healthy
- [ ] Scaling capacity review
  - Current: 5 concurrent users stable
  - Growing to 10+? See OPERATIONAL_RUNBOOK.md scaling section
- [ ] Security audit
  - [ ] JWT_SECRET_KEY still secure
  - [ ] LDAP credentials (if used) still valid
  - [ ] API logs show no unauthorized access attempts
- [ ] Documentation review
  - [ ] Runbooks match current deployment
  - [ ] Troubleshooting section covers recent issues

## Alert Thresholds

| Metric | Warning | Critical | Action |
|--------|---------|----------|--------|
| CPU (backend) | >50% | >70% | Restart, investigate slow queries |
| Memory (backend) | >400MB | >600MB | Restart, check for memory leak |
| Disk (backups) | >10GB | >15GB | Delete old backups, increase retention |
| API response (p95) | >500ms | >1s | Check slow query logs |
| Backup age | >36 hours | >48 hours | Check cron, manual run required |
| Database locked | 1 event/week | 5+ events/week | Investigate, may need upgrade |

## Quick Troubleshooting

**Service down**
→ Check: `docker-compose ps` → `docker-compose logs SERVICE_NAME` → `docker-compose restart SERVICE_NAME`

**Slow responses**
→ Check: `docker stats` → `docker-compose logs backend | grep "slow"` → Consider vertical scaling

**Database locked**
→ Restart backend: `docker-compose restart backend`

**Out of disk space**
→ Check: `du -sh data/ backups/` → Clean old backups → Extend volume

---

**Print and post near server, or set email reminders for weekly checks.**
```

**Acceptance Criteria**:
- [ ] Daily checklist <5 minutes
- [ ] Weekly checklist <15 minutes
- [ ] Monthly procedure <30 minutes
- [ ] Alert thresholds with clear actions
- [ ] Troubleshooting linked to runbook

---

### Task 6: Disaster Recovery Plan
**File**: `docs/DISASTER_RECOVERY_PLAN.md` (new, ~100 lines)  
**Status**: Ready  
**Description**: Procedures for worst-case failure scenarios

**Content** (~100 lines):
```markdown
# Disaster Recovery Plan

**Objective**: Restore production service within 10 minutes and zero data loss.

---

## Scenarios & Procedures

### Scenario 1: Database Corrupted

**Detection**: Integrity check fails or data unexpectedly missing

**Recovery Steps**:
1. `docker-compose down`
2. `./scripts/restore.sh backups/latest.tar.gz --validate`
3. `docker-compose up -d`
4. Run integrity check: `PRAGMA integrity_check;`
5. Notify users if data loss (max 1 day, in-flight transactions)

**RTO**: <10 minutes  
**RPO**: 1 day

---

### Scenario 2: Complete System Failure (Hardware)

**Detection**: Server doesn't boot or network card failed

**Recovery Steps**:
1. Provision new Ubuntu 22.04 LTS server (same specs)
2. `git clone <repo>` and `cd /opt/tfm-inventory`
3. Restore: `./scripts/restore.sh /path/to/backup.tar.gz --validate`
4. Update DNS/load balancer to new server IP
5. Verify: All services healthy, data present, users can login

**RTO**: <30 minutes (depends on provisioning)  
**RPO**: 1 day

---

### Scenario 3: Data Center Failure

**Detection**: Entire data center unreachable

**Recovery Steps**:
1. **Activate secondary site** (if available) or failover to cloud
2. Clone repository on new server
3. Restore latest backup: `./scripts/restore.sh backup.tar.gz --validate`
4. Update DNS to new location
5. Notify users of 1-day recovery (latest backup)

**RTO**: 30-60 minutes (depends on secondary readiness)  
**RPO**: 1 day

---

## Regular Testing

### Monthly Backup Test

```bash
# Run on staging environment
./scripts/restore.sh backups/production-latest.tar.gz --validate

# Checklist:
- [ ] Restore completes without errors
- [ ] All services start correctly
- [ ] Database passes integrity check
- [ ] 10K+ items present (sanity check)
- [ ] API responds at /health
- [ ] Frontend loads
```

### Quarterly Full Failover Drill

1. Provision new server with same specs as production
2. Restore full backup
3. Run through daily health checks
4. Simulate 5 concurrent users
5. Document any issues and update this plan

---

## Prevention

| Prevention | Implementation |
|-----------|-----------------|
| Offsite backups | Upload weekly backup to S3/cloud storage |
| Multiple AZs | Deploy secondary in different region (future) |
| Monitoring | Alert on service restart, high CPU, disk full |
| Testing | Monthly restore test, quarterly failover drill |

---

## Success Criteria

- [ ] Restore completes in <10 minutes
- [ ] Zero data loss (1-day RPO acceptable)
- [ ] All services healthy post-restore
- [ ] Users can login and access data
- [ ] Monthly test succeeds 100%

---

**Last Updated**: 2026-04-22  
**Next Review**: 2026-05-22  
**Owner**: Operations Team
```

**Acceptance Criteria**:
- [ ] Covers 3+ failure scenarios
- [ ] Clear step-by-step recovery procedures
- [ ] RTO/RPO documented
- [ ] Regular testing schedule
- [ ] Prevention measures listed

---

### Task 7: Documentation Integration & Sign-Off
**File**: `docs/README_OPERATIONS.md` (new, ~70 lines)  
**Status**: Ready  
**Description**: Index and integration guide for all operational docs

**Content** (~70 lines):
```markdown
# Operations Documentation Index

This directory contains everything needed to operate TFM aInventory in production.

## Quick Links

| Document | Purpose | Audience | Time |
|----------|---------|----------|------|
| [DEPLOYMENT_QUICKSTART.md](DEPLOYMENT_QUICKSTART.md) | First-time setup | DevOps/SysAdmin | 5 min |
| [OPERATIONAL_RUNBOOK.md](OPERATIONAL_RUNBOOK.md) | Daily/weekly tasks | Operations team | 5-30 min |
| [HEALTH_MONITORING_CHECKLIST.md](HEALTH_MONITORING_CHECKLIST.md) | Health checks | Site manager | 5 min (daily) |
| [DISASTER_RECOVERY_PLAN.md](DISASTER_RECOVERY_PLAN.md) | Failure recovery | Operations lead | 10 min |
| [PERFORMANCE_BASELINE.md](../PERFORMANCE_BASELINE.md) | System capacity | DevOps | 10 min |
| [LOAD_TEST_GUIDE.md](LOAD_TEST_GUIDE.md) | Performance testing | QA/DevOps | 30 min |

## Typical Workflows

### New Deployment
1. Read: DEPLOYMENT_QUICKSTART.md
2. Run: `./deploy.sh production`
3. Setup: Cron jobs via `config/backup-cron.sh`

### Daily Operations
1. Print/review: HEALTH_MONITORING_CHECKLIST.md
2. Run daily checks (5 min)
3. Review logs: `docker-compose logs | grep ERROR`

### Emergency Incident
1. Consult: DISASTER_RECOVERY_PLAN.md
2. Follow recovery steps for scenario
3. Run validation tests
4. Notify stakeholders

### Capacity Planning
1. Review: PERFORMANCE_BASELINE.md
2. Run: LOAD_TEST_GUIDE.md monthly
3. Track trends vs. baseline
4. Plan scaling 30 days in advance

---

## Operational Metrics

**Current Capacity**: 5 concurrent users, 10K items stable  
**System Specs**: 2GB RAM, 10GB disk (recommended: 4GB/50GB)  
**RTO (Recovery Time)**: <10 minutes  
**RPO (Recovery Point)**: 1 day (daily backups)  
**Backup Retention**: 30 days (daily), 90 days (weekly)  

## Support & Escalation

- Developer issues: dev@example.com
- Operational incidents: ops@example.com
- 24/7 on-call: [phone number]

---

**Last Updated**: 2026-04-22  
**Version**: 1.0.0  
**Maintained By**: Operations Team  
**Next Review**: 2026-05-22
```

**Acceptance Criteria**:
- [ ] Links to all operational documents
- [ ] Clear workflow guidance (new deploy, daily ops, emergencies)
- [ ] Quick reference table with audience and time
- [ ] Current capacity metrics documented
- [ ] Support contact information

---

## Dependencies

**Upstream**:
- Plan 1 (Docker/Deployment) — `deploy.sh` and docker-compose.yml required
- Plan 2 (Scale Testing) — Baseline metrics inform runbook scaling guidance
- Phase 5 complete (all features stable)

**Cross-Plan**: None

**Blocked By**: None

---

## Testing Strategy

### Unit Testing (Standalone)
```bash
# Test backup
./scripts/backup.sh manual
# Verify tarball created and valid
tar -tzf backups/inventory-*.tar.gz | wc -l  # Should list files

# Test restore on staging
docker pull $(docker-compose config | grep image)
docker-compose up -d
./scripts/restore.sh backups/latest.tar.gz --validate
```

### Integration Testing
```bash
# Full cycle on clean system
./deploy.sh production
./scripts/backup.sh manual
# Corrupt data
rm data/inventory.db
# Restore
./scripts/restore.sh backups/latest.tar.gz --validate
# Verify data intact
curl http://localhost:8000/health
```

### Operational Testing
```bash
# Simulate daily health checks
bash << 'EOF'
docker-compose ps
curl http://localhost:8000/health
docker stats --no-stream
EOF

# Monthly backup test
./scripts/backup.sh manual
# On staging: ./scripts/restore.sh
```

---

## Success Metrics

- [ ] Backup script creates valid tarballs
- [ ] Restore recovers full system in <10 minutes
- [ ] Triple-confirmation prevents accidental restore
- [ ] Health checklist completes in <5 minutes
- [ ] Runbook enables new ops to deploy independently
- [ ] Disaster recovery scenarios tested monthly
- [ ] Zero data loss in restore validation
- [ ] All documentation clear and linked

---

## Notes

- Backup strategy: Daily incremental (via DB WAL), weekly full backups
- Cron jobs require `sudo` to install; runs as root to access all files
- Restore requires triple confirmation to prevent accidents
- Operations team should run monthly restore test on staging
- Documentation reviewed and updated quarterly

---

**Effort Estimate**: 20 hours (2-3 days)  
**Dependencies**: Plan 1 complete (deploy.sh and docker-compose)  
**Risk**: Low (documentation + testing, no production code changes)

---

Last updated: 2026-04-22 (Planning Phase)