tfm_ainventory/.planning/phases/06-deployment-scale/PLAN-02-OPERATIONAL-RUNBOOK.md

Phase 6, Plan 3: Backup/Restore & Operational Runbook

---

plan: 06-deployment-scale/03-backup-runbook
feature: Automated backup/restore, disaster recovery, operational runbook
status: Ready for execution
estimated_tasks: 7
total_lines: ~500 (backup script ~120, restore script ~100, runbook ~150, monitoring ~50, troubleshooting ~80)

---

Overview

This plan ensures operational resilience through automated backup/restore and comprehensive documentation. It creates:

1. Backup automation — Daily/weekly scheduled backups of DB, config, certificates
2. Restore validation — Automated restore testing, zero-data-loss confirmation
3. Operational runbook — Step-by-step procedures for deployment, scaling, troubleshooting
4. Disaster recovery — RTO <10 minutes, RPO 1 day (configurable)
5. Health monitoring — Checklist-style monitoring for ops teams

Success: Backup/restore cycle tested and validated; runbook enables new ops to deploy and maintain without support.

---

Tasks

Task 1: Backup Automation Script

File: scripts/backup.sh (new, ~120 lines)
Status: Ready
Description: Automated backup of database, config, and certificates

Content (~120 lines):

#!/bin/bash
set -euo pipefail

# Phase 6, Plan 3, Task 1: Automated Backup Script
# Usage: ./backup.sh [daily|weekly|manual] [retention_days]

BACKUP_TYPE="${1:-manual}"
RETENTION_DAYS="${2:-30}"
BACKUP_DIR="./backups"
DATA_DIR="./data"
TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S)
BACKUP_FILE="$BACKUP_DIR/inventory-$TIMESTAMP.tar.gz"

# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; exit 1; }

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Verify Docker is running
docker ps > /dev/null 2>&1 || log_error "Docker daemon not running"

# 1. Verify services are running
log_info "Checking services..."
if ! docker-compose ps | grep -q "Up"; then
  log_warn "Not all services running; attempting to start..."
  docker-compose up -d
fi

# 2. Create backup tarball
log_info "Creating backup: $BACKUP_FILE"

# Stop backend to ensure DB consistency
log_info "Stopping backend service (DB consistency)..."
docker-compose stop backend

# Wait for graceful shutdown
sleep 2

# Create tarball
tar --exclude='$DATA_DIR/caddy_*' \
    --exclude='$BACKUP_DIR' \
    --exclude='.git' \
    --exclude='node_modules' \
    --exclude='.next' \
    -czf "$BACKUP_FILE" \
    "$DATA_DIR/inventory.db" \
    "$DATA_DIR/inventory.db-wal" \
    "$DATA_DIR/inventory.db-shm" \
    "config/" \
    "inventory.env" \
    2>/dev/null || log_error "Backup creation failed"

# Restart backend
log_info "Restarting backend service..."
docker-compose start backend

# Wait for backend to be ready
sleep 5

# Verify backend is healthy
if curl -sf "http://localhost:8000/health" > /dev/null; then
  log_info "Backend restored and healthy"
else
  log_warn "Backend not responding yet; check logs with 'docker-compose logs -f backend'"
fi

# 3. Verify backup integrity
log_info "Verifying backup integrity..."
if tar -tzf "$BACKUP_FILE" > /dev/null 2>&1; then
  log_info "✓ Backup verified"
else
  log_error "Backup corrupted"
fi

# 4. Log backup metadata
BACKUP_SIZE=$(du -h "$BACKUP_FILE" | cut -f1)
BACKUP_VERSION=$(cat VERSION.json 2>/dev/null | grep version | cut -d'"' -f4 || echo "unknown")
log_info "Backup size: $BACKUP_SIZE, Version: $BACKUP_VERSION"

# 5. Cleanup old backups
log_info "Cleaning up backups older than $RETENTION_DAYS days..."
find "$BACKUP_DIR" -name "inventory-*.tar.gz" -type f -mtime +$RETENTION_DAYS -delete

# 6. Create backup manifest
cat > "$BACKUP_DIR/MANIFEST.txt" << EOF
Backup Metadata
===============
Timestamp: $TIMESTAMP
Type: $BACKUP_TYPE
Retention: $RETENTION_DAYS days
Size: $BACKUP_SIZE
Version: $BACKUP_VERSION
Contents: DB, config, certificates (excluding certs)
Restored: Not yet

Command to restore:
./restore.sh $BACKUP_FILE
EOF

log_info "Backup complete: $BACKUP_FILE"
log_info "Retention policy: Delete after $RETENTION_DAYS days"
log_info "Next backup: $(date -d '+1 day' '+%Y-%m-%d')"

Acceptance Criteria:

• Creates gzip tarball of DB, config, certificates
• Stops backend before backup for consistency
• Restarts backend after backup
• Verifies tarball integrity
• Cleans up old backups based on retention
• Creates manifest with metadata
• Handles errors gracefully

Testing:

chmod +x scripts/backup.sh
./deploy.sh production
./scripts/backup.sh manual
# Verify backup created
ls -lh backups/
# Verify tarball integrity
tar -tzf backups/inventory-*.tar.gz | head

---

Task 2: Restore & Validation Script

File: scripts/restore.sh (new, ~100 lines)
Status: Ready
Description: Restore from backup, validate data integrity

Content (~100 lines):

#!/bin/bash
set -euo pipefail

# Phase 6, Plan 3, Task 2: Restore from Backup
# Usage: ./restore.sh <backup_file> [--validate]

BACKUP_FILE="${1:-}"
VALIDATE="${2:---validate}"

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; exit 1; }

# Validate input
[[ -z "$BACKUP_FILE" ]] && log_error "Usage: ./restore.sh <backup_file>"
[[ ! -f "$BACKUP_FILE" ]] && log_error "Backup file not found: $BACKUP_FILE"

log_info "Restoring from: $BACKUP_FILE"

# Triple confirmation (security requirement)
echo "⚠️  WARNING: This will overwrite current data!"
read -p "Type 'RESTORE' to confirm (1/3): " confirm1
[[ "$confirm1" != "RESTORE" ]] && log_error "Restore cancelled"

read -p "Type 'RESTORE' again to confirm (2/3): " confirm2
[[ "$confirm2" != "RESTORE" ]] && log_error "Restore cancelled"

read -p "Type 'RESTORE' one more time to confirm (3/3): " confirm3
[[ "$confirm3" != "RESTORE" ]] && log_error "Restore cancelled"

log_warn "Proceeding with restore..."

# 1. Stop services
log_info "Stopping services..."
docker-compose down

# 2. Backup current data (safety copy)
log_info "Creating safety backup of current data..."
mkdir -p data/backups_before_restore
tar -czf "data/backups_before_restore/backup-before-restore-$(date +%s).tar.gz" \
    data/inventory.db data/inventory.db-* 2>/dev/null || true

# 3. Extract backup
log_info "Extracting backup..."
tar -xzf "$BACKUP_FILE" -C . || log_error "Backup extraction failed"

# 4. Verify essential files
log_info "Verifying restored files..."
[[ -f "data/inventory.db" ]] || log_error "Database not found in backup"
[[ -f "inventory.env" ]] || log_error "inventory.env not found in backup"

# 5. Restart services
log_info "Restarting services..."
docker-compose up -d

# 6. Wait for health
log_info "Waiting for services to be healthy..."
max_attempts=30
attempt=0
while [[ $attempt -lt $max_attempts ]]; do
  if curl -sf "http://localhost:8000/health" > /dev/null 2>&1; then
    log_info "Services healthy!"
    break
  fi
  attempt=$((attempt + 1))
  sleep 2
done

if [[ $attempt -eq $max_attempts ]]; then
  log_warn "Services did not become healthy within 60 seconds"
  docker-compose logs backend
  exit 1
fi

# 7. Validation tests (if requested)
if [[ "$VALIDATE" == "--validate" ]]; then
  log_info "Running validation tests..."
  
  # Test 1: Database accessible
  DB_ITEMS=$(docker-compose exec backend sqlite3 /app/data/inventory.db \
    "SELECT COUNT(*) FROM items" 2>/dev/null || echo "0")
  log_info "✓ Database has $DB_ITEMS items"
  
  # Test 2: API responsive
  if curl -sf "http://localhost:8000/health" > /dev/null; then
    log_info "✓ API health check passed"
  else
    log_error "API health check failed"
  fi
  
  # Test 3: Frontend loads
  if curl -sf "http://localhost:3000" > /dev/null 2>&1; then
    log_info "✓ Frontend loads"
  else
    log_warn "⚠ Frontend check failed (normal on first startup)"
  fi
fi

log_info "Restore complete!"
log_warn "Remember to verify data in the application before returning to production"

Acceptance Criteria:

• Prompts for triple confirmation
• Creates safety backup before restore
• Extracts tarball with proper permissions
• Validates essential files present
• Restarts all services
• Optional validation tests (DB count, API health, frontend)
• Clear error messages on failure

Testing:

chmod +x scripts/restore.sh
# Create a backup first
./scripts/backup.sh
# Simulate data corruption
rm data/inventory.db
# Restore and validate
./scripts/restore.sh backups/inventory-*.tar.gz --validate
# Verify data returned

---

Task 3: Cron Job Configuration

File: config/backup-cron.sh (new, ~50 lines)
Status: Ready
Description: Setup automated daily/weekly backups via cron

Content (~50 lines):

#!/bin/bash
# Phase 6, Plan 3, Task 3: Install Cron Jobs
# Run with: sudo bash config/backup-cron.sh

DEPLOY_DIR=$(pwd)
CRON_SCHEDULE_DAILY="0 2 * * *"  # 2 AM every day
CRON_SCHEDULE_WEEKLY="0 3 * * 0" # 3 AM every Sunday

# Check if running with sudo
if [[ $EUID -ne 0 ]]; then
  echo "This script must be run with sudo"
  exit 1
fi

echo "Installing cron jobs for automated backups..."

# Install daily backup
(crontab -l 2>/dev/null | grep -v "inventory backup"; \
  echo "$CRON_SCHEDULE_DAILY cd $DEPLOY_DIR && bash scripts/backup.sh daily >> logs/backup-daily.log 2>&1") | \
  crontab -

# Install weekly backup
(crontab -l 2>/dev/null | grep -v "inventory backup"; \
  echo "$CRON_SCHEDULE_WEEKLY cd $DEPLOY_DIR && bash scripts/backup.sh weekly 90 >> logs/backup-weekly.log 2>&1") | \
  crontab -

echo "✓ Cron jobs installed"
echo "  Daily backup: $CRON_SCHEDULE_DAILY (retention: 30 days)"
echo "  Weekly backup: $CRON_SCHEDULE_WEEKLY (retention: 90 days)"
echo ""
echo "View active cron jobs:"
crontab -l | grep backup

Acceptance Criteria:

• Installs daily and weekly cron jobs
• Logs to files for audit trail
• Daily retention 30 days, weekly 90 days
• Can be installed/uninstalled without manual edits
• Works on Ubuntu 22.04+

Testing:

sudo bash config/backup-cron.sh
# Verify installation
sudo crontab -l | grep backup
# Simulate a run
cd /path/to/tfm-inventory && bash scripts/backup.sh daily

---

Task 4: Operational Runbook

File: docs/OPERATIONAL_RUNBOOK.md (new, ~200 lines)
Status: Ready
Description: Step-by-step procedures for ops teams

Content (~200 lines):

# Operational Runbook

**Audience**: Systems operators, site managers, DevOps teams  
**Target**: Minimal training required; step-by-step procedures

---

## 1. Initial Deployment

### Requirements
- Ubuntu 22.04 LTS or similar
- Docker, Docker Compose installed
- 2GB RAM, 10GB disk (recommended: 4GB/50GB for production)
- Internet access (first-time setup only)

### Steps

1. **Clone repository**
   ```bash
   git clone <repo_url> /opt/tfm-inventory
   cd /opt/tfm-inventory

2. Configure environment

   cp inventory.env.template inventory.env
   # Edit inventory.env with your settings:
   # - BACKEND_PORT, FRONTEND_PORT
   # - JWT_SECRET_KEY (generate: openssl rand -hex 32)
   # - AI settings (Gemini/Claude API keys)
   # - LDAP settings (if using enterprise auth)
   

3. Deploy

   chmod +x deploy.sh
   ./deploy.sh production
   

4. Verify

   • Frontend: http://your-server:3000
   • Backend API: http://your-server:8000
   • API Docs: http://your-server:8000/docs

5. Create admin user

   docker-compose exec backend python -c "
   from backend.db import SessionLocal, User
   db = SessionLocal()
   user = User(username='admin', hashed_password='...', is_admin=True)
   db.add(user)
   db.commit()
   "
   

---

2. Daily Operations

Health Checks (Daily)

# Check all services
docker-compose ps
# Expected: All services "Up"

# Check API health
curl http://localhost:8000/health

# Check database size
du -h data/inventory.db

# Check logs for errors
docker-compose logs | grep ERROR

Backup (Automated)

# Verify automatic backup ran
ls -lh backups/ | head -1

# Manual backup (if needed)
./scripts/backup.sh manual

# View backup schedule
sudo crontab -l | grep backup

Monitoring

# Real-time logs
docker-compose logs -f

# Backend performance
docker stats --no-stream | grep backend

# Database status
docker-compose exec backend sqlite3 /app/data/inventory.db \
  "SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"

---

3. Troubleshooting

Service won't start

# Check Docker daemon
docker ps

# Check port conflicts
netstat -tuln | grep 8000

# View service logs
docker-compose logs backend
docker-compose logs frontend
docker-compose logs proxy

High CPU/Memory

# Identify container
docker stats

# Restart container
docker-compose restart backend

# Check for slow queries
docker-compose logs backend | grep "slow query"

Database locked

# Restart backend
docker-compose restart backend

# Check WAL mode status
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA journal_mode;"

HTTPS Certificate issues

# Certificates regenerated automatically
# If issues persist:
rm -rf data/caddy_*
docker-compose restart proxy
# Wait 30 seconds for new certs

---

4. Scaling Operations

Adding Users (5+ concurrent)

Currently configured and tested for 5 concurrent users safely.

To support more users:

1. Increase backend memory: Edit docker-compose.yml

   backend:
     mem_limit: 4g
   

2. Increase database connections:

   docker-compose exec backend \
     python -c "import backend.config; print(backend.config.DB_POOL_SIZE)"
   

3. Add read replicas (if needed, v3 feature)

Database Growth (10K+ items)

As database grows beyond 10K items:

1. Monitor query performance: PRAGMA optimize;
2. Create indexes on frequently searched columns
3. Vacuum database: VACUUM;

---

5. Backup & Restore

Automated Backups

# Cron jobs run automatically
# Daily: 2 AM, retention 30 days
# Weekly: 3 AM Sundays, retention 90 days

# Verify cron installation
sudo bash config/backup-cron.sh

# View backup history
ls -lh backups/

Manual Restore

# List available backups
ls backups/

# Restore specific backup
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz

# Validate data after restore
curl http://localhost:8000/health

RTO (Recovery Time Objective): <10 minutes
RPO (Recovery Point Objective): 1 day (daily backup)

---

6. Disaster Recovery

Complete System Failure

1. Restore on new server

   # Fresh Ubuntu 22.04
   sudo apt-get update && sudo apt-get install -y docker.io docker-compose
   git clone <repo> /opt/tfm-inventory
   cd /opt/tfm-inventory
   ./scripts/restore.sh backups/latest.tar.gz --validate
   

2. Verify data integrity

   curl http://localhost:8000/health
   # Check item count in database
   

3. Return to production

   • Update DNS/load balancer
   • Notify users

Data Corruption

1. Investigate

   docker-compose exec backend python -c "
   import sqlite3
   db = sqlite3.connect('/app/data/inventory.db')
   # Run integrity check
   print(db.execute('PRAGMA integrity_check').fetchall())
   "
   

2. If corrupted

   • Restore from last backup: ./scripts/restore.sh backups/latest.tar.gz
   • Notify affected users of recovery

---

7. Updates & Upgrades

Patch Update (v1.14.x → v1.14.y)

# Backup first
./scripts/backup.sh manual

# Pull latest code
git pull origin main

# Rebuild and restart
./deploy.sh production --rebuild

# Verify
curl http://localhost:8000/health

Major Update (v1.x → v2.x)

# Create backup before proceeding
./scripts/backup.sh manual

# Review CHANGELOG for breaking changes
cat CHANGELOG.md | grep "v2.0"

# Follow upgrade guide
cat docs/UPGRADE_GUIDE.md

# Test in staging first
./scripts/restore.sh backups/production.tar.gz
# (Test on staging environment)

# Proceed to production
git checkout v2.0
./deploy.sh production --rebuild

---

8. Emergency Contacts

• Developer Support: dev@example.com
• Infrastructure: ops@example.com
• 24/7 On-call: [contact info]

---

Last Updated: 2026-04-22
Version: 1.0
Maintainer: Operations Team

**Acceptance Criteria**:
- [ ] Covers full deployment lifecycle
- [ ] Health check procedures documented
- [ ] Troubleshooting section covers common issues
- [ ] Scaling guidance clear
- [ ] Backup/restore procedures step-by-step
- [ ] Written for non-technical audience
- [ ] Emergency contacts and escalation paths

---

### Task 5: Health Monitoring Checklist
**File**: `docs/HEALTH_MONITORING_CHECKLIST.md` (new, ~80 lines)  
**Status**: Ready  
**Description**: Daily/weekly health checks for ops teams

**Content** (~80 lines):
```markdown
# Health Monitoring Checklist

Use this checklist for daily/weekly health reviews.

## Daily (5 minutes)

- [ ] All services running: `docker-compose ps`
  - Expected: backend, frontend, proxy all "Up"
- [ ] API responsive: `curl http://localhost:8000/health`
  - Expected: 200 OK, response <100ms
- [ ] Frontend loads: `curl http://localhost:3000/`
  - Expected: 200 OK
- [ ] Recent errors in logs: `docker-compose logs | grep ERROR | tail -5`
  - Action: Investigate any ERROR-level logs
- [ ] Database accessible: `docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"`
  - Action: If fails, restart backend

## Weekly (15 minutes)

- [ ] Backup completed: `ls -lh backups/ | head -1`
  - Check timestamp is within last 24 hours
- [ ] Disk usage: `du -sh data/ config/ backups/`
  - Expected: data/ <5GB, backups/ <10GB (5 weeks @ 2GB/week)
  - Action: If backups >10GB, verify cron retention is set correctly
- [ ] Database size: `docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT page_count * page_size / (1024*1024) FROM pragma_page_count(), pragma_page_size();"`
  - Action: If >1GB, consider optimization
- [ ] Service resource usage: `docker stats --no-stream`
  - Expected: backend <70% CPU, <500MB RAM
  - Action: If exceeds, investigate slow queries
- [ ] Restore test: `./scripts/backup.sh manual`
  - Action: Run monthly
- [ ] Update check: `git status`
  - Action: Review available updates

## Monthly (30 minutes)

- [ ] Restore from backup test
  ```bash
  # On staging environment
  ./scripts/restore.sh backups/latest.tar.gz --validate

• Action: Confirm zero data loss, all services healthy
• Scaling capacity review
  • Current: 5 concurrent users stable
  • Growing to 10+? See OPERATIONAL_RUNBOOK.md scaling section
• Security audit
  • JWT_SECRET_KEY still secure
  • LDAP credentials (if used) still valid
  • API logs show no unauthorized access attempts
• Documentation review
  • Runbooks match current deployment
  • Troubleshooting section covers recent issues

Alert Thresholds

Metric	Warning	Critical	Action
CPU (backend)	>50%	>70%	Restart, investigate slow queries
Memory (backend)	>400MB	>600MB	Restart, check for memory leak
Disk (backups)	>10GB	>15GB	Delete old backups, increase retention
API response (p95)	>500ms	>1s	Check slow query logs
Backup age	>36 hours	>48 hours	Check cron, manual run required
Database locked	1 event/week	5+ events/week	Investigate, may need upgrade

Quick Troubleshooting

Service down → Check: docker-compose ps → docker-compose logs SERVICE_NAME → docker-compose restart SERVICE_NAME

Slow responses → Check: docker stats → docker-compose logs backend | grep "slow" → Consider vertical scaling

Database locked → Restart backend: docker-compose restart backend

Out of disk space → Check: du -sh data/ backups/ → Clean old backups → Extend volume

---

Print and post near server, or set email reminders for weekly checks.

**Acceptance Criteria**:
- [ ] Daily checklist <5 minutes
- [ ] Weekly checklist <15 minutes
- [ ] Monthly procedure <30 minutes
- [ ] Alert thresholds with clear actions
- [ ] Troubleshooting linked to runbook

---

### Task 6: Disaster Recovery Plan
**File**: `docs/DISASTER_RECOVERY_PLAN.md` (new, ~100 lines)  
**Status**: Ready  
**Description**: Procedures for worst-case failure scenarios

**Content** (~100 lines):
```markdown
# Disaster Recovery Plan

**Objective**: Restore production service within 10 minutes and zero data loss.

---

## Scenarios & Procedures

### Scenario 1: Database Corrupted

**Detection**: Integrity check fails or data unexpectedly missing

**Recovery Steps**:
1. `docker-compose down`
2. `./scripts/restore.sh backups/latest.tar.gz --validate`
3. `docker-compose up -d`
4. Run integrity check: `PRAGMA integrity_check;`
5. Notify users if data loss (max 1 day, in-flight transactions)

**RTO**: <10 minutes  
**RPO**: 1 day

---

### Scenario 2: Complete System Failure (Hardware)

**Detection**: Server doesn't boot or network card failed

**Recovery Steps**:
1. Provision new Ubuntu 22.04 LTS server (same specs)
2. `git clone <repo>` and `cd /opt/tfm-inventory`
3. Restore: `./scripts/restore.sh /path/to/backup.tar.gz --validate`
4. Update DNS/load balancer to new server IP
5. Verify: All services healthy, data present, users can login

**RTO**: <30 minutes (depends on provisioning)  
**RPO**: 1 day

---

### Scenario 3: Data Center Failure

**Detection**: Entire data center unreachable

**Recovery Steps**:
1. **Activate secondary site** (if available) or failover to cloud
2. Clone repository on new server
3. Restore latest backup: `./scripts/restore.sh backup.tar.gz --validate`
4. Update DNS to new location
5. Notify users of 1-day recovery (latest backup)

**RTO**: 30-60 minutes (depends on secondary readiness)  
**RPO**: 1 day

---

## Regular Testing

### Monthly Backup Test

```bash
# Run on staging environment
./scripts/restore.sh backups/production-latest.tar.gz --validate

# Checklist:
- [ ] Restore completes without errors
- [ ] All services start correctly
- [ ] Database passes integrity check
- [ ] 10K+ items present (sanity check)
- [ ] API responds at /health
- [ ] Frontend loads

Quarterly Full Failover Drill

1. Provision new server with same specs as production
2. Restore full backup
3. Run through daily health checks
4. Simulate 5 concurrent users
5. Document any issues and update this plan

---

Prevention

Prevention	Implementation
Offsite backups	Upload weekly backup to S3/cloud storage
Multiple AZs	Deploy secondary in different region (future)
Monitoring	Alert on service restart, high CPU, disk full
Testing	Monthly restore test, quarterly failover drill

---

Success Criteria

• Restore completes in <10 minutes
• Zero data loss (1-day RPO acceptable)
• All services healthy post-restore
• Users can login and access data
• Monthly test succeeds 100%

---

Last Updated: 2026-04-22
Next Review: 2026-05-22
Owner: Operations Team

**Acceptance Criteria**:
- [ ] Covers 3+ failure scenarios
- [ ] Clear step-by-step recovery procedures
- [ ] RTO/RPO documented
- [ ] Regular testing schedule
- [ ] Prevention measures listed

---

### Task 7: Documentation Integration & Sign-Off
**File**: `docs/README_OPERATIONS.md` (new, ~70 lines)  
**Status**: Ready  
**Description**: Index and integration guide for all operational docs

**Content** (~70 lines):
```markdown
# Operations Documentation Index

This directory contains everything needed to operate TFM aInventory in production.

## Quick Links

| Document | Purpose | Audience | Time |
|----------|---------|----------|------|
| [DEPLOYMENT_QUICKSTART.md](DEPLOYMENT_QUICKSTART.md) | First-time setup | DevOps/SysAdmin | 5 min |
| [OPERATIONAL_RUNBOOK.md](OPERATIONAL_RUNBOOK.md) | Daily/weekly tasks | Operations team | 5-30 min |
| [HEALTH_MONITORING_CHECKLIST.md](HEALTH_MONITORING_CHECKLIST.md) | Health checks | Site manager | 5 min (daily) |
| [DISASTER_RECOVERY_PLAN.md](DISASTER_RECOVERY_PLAN.md) | Failure recovery | Operations lead | 10 min |
| [PERFORMANCE_BASELINE.md](../PERFORMANCE_BASELINE.md) | System capacity | DevOps | 10 min |
| [LOAD_TEST_GUIDE.md](LOAD_TEST_GUIDE.md) | Performance testing | QA/DevOps | 30 min |

## Typical Workflows

### New Deployment
1. Read: DEPLOYMENT_QUICKSTART.md
2. Run: `./deploy.sh production`
3. Setup: Cron jobs via `config/backup-cron.sh`

### Daily Operations
1. Print/review: HEALTH_MONITORING_CHECKLIST.md
2. Run daily checks (5 min)
3. Review logs: `docker-compose logs | grep ERROR`

### Emergency Incident
1. Consult: DISASTER_RECOVERY_PLAN.md
2. Follow recovery steps for scenario
3. Run validation tests
4. Notify stakeholders

### Capacity Planning
1. Review: PERFORMANCE_BASELINE.md
2. Run: LOAD_TEST_GUIDE.md monthly
3. Track trends vs. baseline
4. Plan scaling 30 days in advance

---

## Operational Metrics

**Current Capacity**: 5 concurrent users, 10K items stable  
**System Specs**: 2GB RAM, 10GB disk (recommended: 4GB/50GB)  
**RTO (Recovery Time)**: <10 minutes  
**RPO (Recovery Point)**: 1 day (daily backups)  
**Backup Retention**: 30 days (daily), 90 days (weekly)  

## Support & Escalation

- Developer issues: dev@example.com
- Operational incidents: ops@example.com
- 24/7 on-call: [phone number]

---

**Last Updated**: 2026-04-22  
**Version**: 1.0.0  
**Maintained By**: Operations Team  
**Next Review**: 2026-05-22

Acceptance Criteria:

• Links to all operational documents
• Clear workflow guidance (new deploy, daily ops, emergencies)
• Quick reference table with audience and time
• Current capacity metrics documented
• Support contact information

---

Dependencies

Upstream:

• Plan 1 (Docker/Deployment) — deploy.sh and docker-compose.yml required
• Plan 2 (Scale Testing) — Baseline metrics inform runbook scaling guidance
• Phase 5 complete (all features stable)

Cross-Plan: None

Blocked By: None

---

Testing Strategy

Unit Testing (Standalone)

# Test backup
./scripts/backup.sh manual
# Verify tarball created and valid
tar -tzf backups/inventory-*.tar.gz | wc -l  # Should list files

# Test restore on staging
docker pull $(docker-compose config | grep image)
docker-compose up -d
./scripts/restore.sh backups/latest.tar.gz --validate

Integration Testing

# Full cycle on clean system
./deploy.sh production
./scripts/backup.sh manual
# Corrupt data
rm data/inventory.db
# Restore
./scripts/restore.sh backups/latest.tar.gz --validate
# Verify data intact
curl http://localhost:8000/health

Operational Testing

# Simulate daily health checks
bash << 'EOF'
docker-compose ps
curl http://localhost:8000/health
docker stats --no-stream
EOF

# Monthly backup test
./scripts/backup.sh manual
# On staging: ./scripts/restore.sh

---

Success Metrics

• Backup script creates valid tarballs
• Restore recovers full system in <10 minutes
• Triple-confirmation prevents accidental restore
• Health checklist completes in <5 minutes
• Runbook enables new ops to deploy independently
• Disaster recovery scenarios tested monthly
• Zero data loss in restore validation
• All documentation clear and linked

---

Notes

• Backup strategy: Daily incremental (via DB WAL), weekly full backups
• Cron jobs require sudo to install; runs as root to access all files
• Restore requires triple confirmation to prevent accidents
• Operations team should run monthly restore test on staging
• Documentation reviewed and updated quarterly

---

Effort Estimate: 20 hours (2-3 days)
Dependencies: Plan 1 complete (deploy.sh and docker-compose)
Risk: Low (documentation + testing, no production code changes)

---

Last updated: 2026-04-22 (Planning Phase)