# Operational Runbook

**Audience**: Systems operators, site managers, DevOps teams  
**Target**: Minimal training required; step-by-step procedures  
**Last Updated**: 2026-04-22

---

## Overview

This runbook covers operational procedures for both Docker and Standalone deployment modes. Both modes use the same configuration files (inventory.env) and backup/restore scripts.

---

## 1. Initial Deployment

### Requirements

- Ubuntu 22.04 LTS or similar
- For Docker mode: Docker and Docker Compose installed
- For Standalone mode: Python 3.12+, Node.js 18+, npm
- 2GB RAM minimum, 10GB disk (recommended: 4GB/50GB for production)
- Internet access (first-time setup only)

### Docker Deployment Steps

1. **Clone repository**
   ```bash
   git clone <repo_url> /opt/tfm-inventory
   cd /opt/tfm-inventory
   ```

2. **Configure environment**
   ```bash
   cp inventory.env.template inventory.env
   # Edit inventory.env with your settings:
   # - BACKEND_PORT=8000, FRONTEND_PORT=3000
   # - JWT_SECRET_KEY (generate: openssl rand -hex 32)
   # - AI settings (Gemini/Claude API keys, optional)
   # - LDAP settings (if using enterprise auth)
   ```

3. **Deploy**
   ```bash
   chmod +x deploy.sh scripts/backup.sh scripts/restore.sh
   ./deploy.sh production
   ```

4. **Verify deployment**
   - Frontend: http://your-server:3000
   - Backend API: http://your-server:8000
   - API Docs: http://your-server:8000/docs
   - Health check: `curl http://localhost:8000/health`

5. **Create admin user** (if not auto-created)
   ```bash
   docker-compose exec backend python -c "
   from backend.db import SessionLocal, User
   db = SessionLocal()
   user = User(username='admin', hashed_password='...', is_admin=True)
   db.add(user)
   db.commit()
   "
   ```

### Standalone Deployment Steps

1. **Clone repository**
   ```bash
   git clone <repo_url> /opt/tfm-inventory
   cd /opt/tfm-inventory
   ```

2. **Configure environment**
   ```bash
   cp inventory.env.template inventory.env
   # Edit with same settings as Docker mode
   ```

3. **Install dependencies**
   ```bash
   # Backend
   cd backend
   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   cd ..

   # Frontend
   cd frontend
   npm install
   cd ..
   ```

4. **Deploy**
   ```bash
   chmod +x start_server.sh scripts/backup.sh scripts/restore.sh
   ./start_server.sh
   ```

5. **Verify deployment**
   - Frontend: http://localhost:3000
   - Backend API: http://localhost:8000
   - Health check: `curl http://localhost:8000/health`

---

## 2. Daily Operations

### Health Checks (Daily, ~5 minutes)

**Docker mode:**
```bash
# Check all services
docker-compose ps
# Expected: All services "Up"

# Check API health
curl http://localhost:8000/health
# Expected: 200 OK, response time <100ms

# Check database
du -h data/inventory.db

# Check for errors
docker-compose logs | grep ERROR | tail -5
```

**Standalone mode:**
```bash
# Check processes
ps aux | grep -E "(uvicorn|next)" | grep -v grep

# Check API health
curl http://localhost:8000/health

# Check database
du -h data/inventory.db

# Check logs
tail -50 logs/backend.log logs/frontend.log 2>/dev/null
```

### Backup (Automated)

```bash
# Verify automatic backup ran (cron jobs)
ls -lh backups/ | head -1
# Expected: File timestamp within last 24 hours

# Manual backup (if needed)
./scripts/backup.sh manual

# View backup schedule
sudo crontab -l | grep backup
```

### Monitoring

**Docker mode:**
```bash
# Real-time logs
docker-compose logs -f

# Backend performance
docker stats --no-stream | grep backend

# Database status
docker-compose exec backend sqlite3 /app/data/inventory.db \
  "SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"
```

**Standalone mode:**
```bash
# Real-time logs
tail -f logs/backend.log logs/frontend.log

# System resources
top -p $(pgrep -f uvicorn | head -1)

# Database status
sqlite3 data/inventory.db \
  "SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"
```

---

## 3. Troubleshooting

### Service Won't Start

**Docker mode:**
```bash
# Check Docker daemon
docker ps

# Check port conflicts
netstat -tuln | grep -E "8000|3000|8906|8907"

# View service logs
docker-compose logs backend
docker-compose logs frontend
docker-compose logs proxy
```

**Standalone mode:**
```bash
# Check if processes are running
ps aux | grep -E "(uvicorn|next)"

# Check port conflicts
netstat -tuln | grep -E "8000|3000"

# Check logs
cat logs/backend.log | tail -50
```

### High CPU/Memory

**Docker mode:**
```bash
# Identify container
docker stats --no-stream

# Restart container
docker-compose restart backend

# Check for slow queries
docker-compose logs backend | grep "slow"
```

**Standalone mode:**
```bash
# Kill and restart
pkill -f uvicorn
pkill -f "next start"
sleep 2
./start_server.sh
```

### Database Locked

**Docker mode:**
```bash
docker-compose restart backend
# Wait 30 seconds
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA journal_mode;"
```

**Standalone mode:**
```bash
pkill -f uvicorn
sleep 2
# Restart backend only (no need for frontend restart)
cd backend && source venv/bin/activate && uvicorn main:app --host 0.0.0.0 --port 8000 &
```

### HTTPS Certificate Issues

**Docker mode:**
```bash
# Certificates regenerated automatically
# If issues persist:
rm -rf data/caddy_*
docker-compose restart proxy
# Wait 30 seconds for new certs to generate
```

**Standalone mode:**
```bash
# For local development/testing, HTTP is sufficient
# For production HTTPS, configure reverse proxy (nginx/Caddy) separately
```

---

## 4. Backup & Restore

### Automated Backups

```bash
# Verify cron jobs are installed
sudo bash config/backup-cron.sh

# View backup history
ls -lh backups/

# Check backup log
tail -20 logs/backup-daily.log
```

**Backup Schedule:**
- Daily: 2 AM, retention 30 days
- Weekly: 3 AM Sundays, retention 90 days

### Manual Backup

```bash
# Create backup
./scripts/backup.sh manual

# Verify backup created and is valid
tar -tzf backups/inventory-*.tar.gz | head
```

### Manual Restore

```bash
# List available backups
ls backups/

# Restore specific backup (Docker mode)
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz --validate

# Restore specific backup (Standalone mode)
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz
# Then restart: ./start_server.sh

# Validate data after restore
curl http://localhost:8000/health
```

**Recovery Objectives:**
- RTO (Recovery Time): <10 minutes
- RPO (Recovery Point): 1 day (daily backup)

---

## 5. Disaster Recovery

### Complete System Failure (Hardware or Data Corruption)

**For Docker:**
1. Provision new Ubuntu 22.04 LTS server (same specs)
2. Clone repository: `git clone <repo> /opt/tfm-inventory && cd /opt/tfm-inventory`
3. Copy latest backup from offsite or previous backup directory
4. Restore: `./scripts/restore.sh /path/to/backup.tar.gz --validate`
5. Update DNS/load balancer to new server IP
6. Verify all services healthy and data present

**For Standalone:**
1. Provision new Ubuntu 22.04 LTS server
2. Clone repository
3. Install dependencies (Python venv, Node.js)
4. Copy latest backup
5. Restore: `./scripts/restore.sh /path/to/backup.tar.gz`
6. Start services: `./start_server.sh`
7. Verify connectivity and data

**RTO**: <30 minutes (provisioning + restore)  
**RPO**: 1 day (latest backup)

### Data Center Failure

1. Activate secondary site or failover to cloud
2. Clone repository on new infrastructure
3. Restore latest backup: `./scripts/restore.sh backup.tar.gz --validate`
4. Update DNS to new location
5. Notify users of recovery (1-day data loss acceptable)

---

## 6. Scaling Operations

### Adding Users (5+ concurrent)

Current configuration supports 5 concurrent users safely.

**Docker mode:**
```bash
# Increase backend memory
# Edit docker-compose.yml:
# backend:
#   mem_limit: 4g

# Increase database connections
docker-compose exec backend \
  python -c "import backend.config; print(backend.config.DB_POOL_SIZE)"
```

**Standalone mode:**
```bash
# Increase Python process resources
# Edit start_server.sh to add workers/processes if using Gunicorn

# Monitor memory usage
ps aux | grep uvicorn
```

### Database Growth (10K+ items)

As inventory grows beyond 10K items:
1. Monitor query performance: `PRAGMA optimize;`
2. Create indexes on frequently searched columns
3. Vacuum database: `VACUUM;`
4. Consider archiving old audit logs (v3 feature)

---

## 7. Updates & Upgrades

### Patch Update (v1.14.x → v1.14.y)

```bash
# Backup first
./scripts/backup.sh manual

# Pull latest code
git pull origin main

# Docker mode:
./deploy.sh production --rebuild

# Standalone mode:
pkill -f uvicorn
pkill -f "next start"
cd backend && source venv/bin/activate && pip install -r requirements.txt
cd ../frontend && npm install && npm run build
./start_server.sh

# Verify
curl http://localhost:8000/health
```

### Major Update (v1.x → v2.x)

```bash
# Create backup before proceeding
./scripts/backup.sh manual

# Review CHANGELOG for breaking changes
cat CHANGELOG.md | grep "v2.0"

# Test in staging first (restore backup there)
./scripts/restore.sh backups/production.tar.gz

# If staging successful, proceed to production
git checkout v2.0
./deploy.sh production --rebuild  # Docker mode
# OR
./start_server.sh  # Standalone mode
```

---

## 8. Performance Baseline

- Backend: <100ms API response time at 5 concurrent users
- Frontend: <1s page load
- Database: <500 queries/min with 10K items
- Memory: Backend <500MB, Frontend <200MB
- CPU: Both services <70% usage under normal load

---

## 9. Emergency Contacts & Escalation

- **Developer Support**: dev@example.com
- **Infrastructure**: ops@example.com
- **24/7 On-call**: [contact info]

---

## Appendix: Quick Reference

| Task | Docker Command | Standalone Command |
|------|---|---|
| Health Check | `docker-compose ps` | `ps aux \| grep -E "(uvicorn\|next)"` |
| View Logs | `docker-compose logs -f` | `tail -f logs/*.log` |
| Restart Backend | `docker-compose restart backend` | `pkill -f uvicorn; ./start_server.sh` |
| Backup | `./scripts/backup.sh manual` | `./scripts/backup.sh manual` |
| Restore | `./scripts/restore.sh file.tar.gz` | `./scripts/restore.sh file.tar.gz` |
| Stop Services | `docker-compose down` | `pkill -f uvicorn; pkill -f "next"` |

---

**Version**: 1.0  
**Last Updated**: 2026-04-22  
**Maintained By**: Operations Team  
**Next Review**: 2026-05-22