tfm_ainventory/docs/OPERATIONAL_RUNBOOK.md

Operational Runbook

Audience: Systems operators, site managers, DevOps teams
Target: Minimal training required; step-by-step procedures
Last Updated: 2026-04-22

---

Overview

This runbook covers operational procedures for both Docker and Standalone deployment modes. Both modes use the same configuration files (inventory.env) and backup/restore scripts.

---

1. Initial Deployment

Requirements

• Ubuntu 22.04 LTS or similar
• For Docker mode: Docker and Docker Compose installed
• For Standalone mode: Python 3.12+, Node.js 18+, npm
• 2GB RAM minimum, 10GB disk (recommended: 4GB/50GB for production)
• Internet access (first-time setup only)

Docker Deployment Steps

1. Clone repository

   git clone <repo_url> /opt/tfm-inventory
   cd /opt/tfm-inventory
   

2. Configure environment

   cp inventory.env.template inventory.env
   # Edit inventory.env with your settings:
   # - BACKEND_PORT=8000, FRONTEND_PORT=3000
   # - JWT_SECRET_KEY (generate: openssl rand -hex 32)
   # - AI settings (Gemini/Claude API keys, optional)
   # - LDAP settings (if using enterprise auth)
   

3. Deploy

   chmod +x deploy.sh scripts/backup.sh scripts/restore.sh
   ./deploy.sh production
   

4. Verify deployment

   • Frontend: http://your-server:3000
   • Backend API: http://your-server:8000
   • API Docs: http://your-server:8000/docs
   • Health check: curl http://localhost:8000/health

5. Create admin user (if not auto-created)

   docker-compose exec backend python -c "
   from backend.db import SessionLocal, User
   db = SessionLocal()
   user = User(username='admin', hashed_password='...', is_admin=True)
   db.add(user)
   db.commit()
   "
   

Standalone Deployment Steps

1. Clone repository

   git clone <repo_url> /opt/tfm-inventory
   cd /opt/tfm-inventory
   

2. Configure environment

   cp inventory.env.template inventory.env
   # Edit with same settings as Docker mode
   

3. Install dependencies

   # Backend
   cd backend
   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   cd ..
   
   # Frontend
   cd frontend
   npm install
   cd ..
   

4. Deploy

   chmod +x start_server.sh scripts/backup.sh scripts/restore.sh
   ./start_server.sh
   

5. Verify deployment

   • Frontend: http://localhost:3000
   • Backend API: http://localhost:8000
   • Health check: curl http://localhost:8000/health

---

2. Daily Operations

Health Checks (Daily, ~5 minutes)

Docker mode:

# Check all services
docker-compose ps
# Expected: All services "Up"

# Check API health
curl http://localhost:8000/health
# Expected: 200 OK, response time <100ms

# Check database
du -h data/inventory.db

# Check for errors
docker-compose logs | grep ERROR | tail -5

Standalone mode:

# Check processes
ps aux | grep -E "(uvicorn|next)" | grep -v grep

# Check API health
curl http://localhost:8000/health

# Check database
du -h data/inventory.db

# Check logs
tail -50 logs/backend.log logs/frontend.log 2>/dev/null

Backup (Automated)

# Verify automatic backup ran (cron jobs)
ls -lh backups/ | head -1
# Expected: File timestamp within last 24 hours

# Manual backup (if needed)
./scripts/backup.sh manual

# View backup schedule
sudo crontab -l | grep backup

Monitoring

Docker mode:

# Real-time logs
docker-compose logs -f

# Backend performance
docker stats --no-stream | grep backend

# Database status
docker-compose exec backend sqlite3 /app/data/inventory.db \
  "SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"

Standalone mode:

# Real-time logs
tail -f logs/backend.log logs/frontend.log

# System resources
top -p $(pgrep -f uvicorn | head -1)

# Database status
sqlite3 data/inventory.db \
  "SELECT COUNT(*) as item_count, SUM(quantity) as total_qty FROM items;"

---

3. Troubleshooting

Service Won't Start

Docker mode:

# Check Docker daemon
docker ps

# Check port conflicts
netstat -tuln | grep -E "8000|3000|8906|8907"

# View service logs
docker-compose logs backend
docker-compose logs frontend
docker-compose logs proxy

Standalone mode:

# Check if processes are running
ps aux | grep -E "(uvicorn|next)"

# Check port conflicts
netstat -tuln | grep -E "8000|3000"

# Check logs
cat logs/backend.log | tail -50

High CPU/Memory

Docker mode:

# Identify container
docker stats --no-stream

# Restart container
docker-compose restart backend

# Check for slow queries
docker-compose logs backend | grep "slow"

Standalone mode:

# Kill and restart
pkill -f uvicorn
pkill -f "next start"
sleep 2
./start_server.sh

Database Locked

Docker mode:

docker-compose restart backend
# Wait 30 seconds
docker-compose exec backend sqlite3 /app/data/inventory.db "PRAGMA journal_mode;"

Standalone mode:

pkill -f uvicorn
sleep 2
# Restart backend only (no need for frontend restart)
cd backend && source venv/bin/activate && uvicorn main:app --host 0.0.0.0 --port 8000 &

HTTPS Certificate Issues

Docker mode:

# Certificates regenerated automatically
# If issues persist:
rm -rf data/caddy_*
docker-compose restart proxy
# Wait 30 seconds for new certs to generate

Standalone mode:

# For local development/testing, HTTP is sufficient
# For production HTTPS, configure reverse proxy (nginx/Caddy) separately

---

4. Backup & Restore

Automated Backups

# Verify cron jobs are installed
sudo bash config/backup-cron.sh

# View backup history
ls -lh backups/

# Check backup log
tail -20 logs/backup-daily.log

Backup Schedule:

• Daily: 2 AM, retention 30 days
• Weekly: 3 AM Sundays, retention 90 days

Manual Backup

# Create backup
./scripts/backup.sh manual

# Verify backup created and is valid
tar -tzf backups/inventory-*.tar.gz | head

Manual Restore

# List available backups
ls backups/

# Restore specific backup (Docker mode)
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz --validate

# Restore specific backup (Standalone mode)
./scripts/restore.sh backups/inventory-2026-04-22_14-30-15.tar.gz
# Then restart: ./start_server.sh

# Validate data after restore
curl http://localhost:8000/health

Recovery Objectives:

• RTO (Recovery Time): <10 minutes
• RPO (Recovery Point): 1 day (daily backup)

---

5. Disaster Recovery

Complete System Failure (Hardware or Data Corruption)

For Docker:

1. Provision new Ubuntu 22.04 LTS server (same specs)
2. Clone repository: git clone <repo> /opt/tfm-inventory && cd /opt/tfm-inventory
3. Copy latest backup from offsite or previous backup directory
4. Restore: ./scripts/restore.sh /path/to/backup.tar.gz --validate
5. Update DNS/load balancer to new server IP
6. Verify all services healthy and data present

For Standalone:

1. Provision new Ubuntu 22.04 LTS server
2. Clone repository
3. Install dependencies (Python venv, Node.js)
4. Copy latest backup
5. Restore: ./scripts/restore.sh /path/to/backup.tar.gz
6. Start services: ./start_server.sh
7. Verify connectivity and data

RTO: <30 minutes (provisioning + restore)
RPO: 1 day (latest backup)

Data Center Failure

1. Activate secondary site or failover to cloud
2. Clone repository on new infrastructure
3. Restore latest backup: ./scripts/restore.sh backup.tar.gz --validate
4. Update DNS to new location
5. Notify users of recovery (1-day data loss acceptable)

---

6. Scaling Operations

Adding Users (5+ concurrent)

Current configuration supports 5 concurrent users safely.

Docker mode:

# Increase backend memory
# Edit docker-compose.yml:
# backend:
#   mem_limit: 4g

# Increase database connections
docker-compose exec backend \
  python -c "import backend.config; print(backend.config.DB_POOL_SIZE)"

Standalone mode:

# Increase Python process resources
# Edit start_server.sh to add workers/processes if using Gunicorn

# Monitor memory usage
ps aux | grep uvicorn

Database Growth (10K+ items)

As inventory grows beyond 10K items:

1. Monitor query performance: PRAGMA optimize;
2. Create indexes on frequently searched columns
3. Vacuum database: VACUUM;
4. Consider archiving old audit logs (v3 feature)

---

7. Updates & Upgrades

Patch Update (v1.14.x → v1.14.y)

# Backup first
./scripts/backup.sh manual

# Pull latest code
git pull origin main

# Docker mode:
./deploy.sh production --rebuild

# Standalone mode:
pkill -f uvicorn
pkill -f "next start"
cd backend && source venv/bin/activate && pip install -r requirements.txt
cd ../frontend && npm install && npm run build
./start_server.sh

# Verify
curl http://localhost:8000/health

Major Update (v1.x → v2.x)

# Create backup before proceeding
./scripts/backup.sh manual

# Review CHANGELOG for breaking changes
cat CHANGELOG.md | grep "v2.0"

# Test in staging first (restore backup there)
./scripts/restore.sh backups/production.tar.gz

# If staging successful, proceed to production
git checkout v2.0
./deploy.sh production --rebuild  # Docker mode
# OR
./start_server.sh  # Standalone mode

---

8. Performance Baseline

• Backend: <100ms API response time at 5 concurrent users
• Frontend: <1s page load
• Database: <500 queries/min with 10K items
• Memory: Backend <500MB, Frontend <200MB
• CPU: Both services <70% usage under normal load

---

9. Emergency Contacts & Escalation

• Developer Support: dev@example.com
• Infrastructure: ops@example.com
• 24/7 On-call: [contact info]

---

Appendix: Quick Reference

Task	Docker Command	Standalone Command
Health Check	docker-compose ps	ps aux | grep -E "(uvicorn|next)"
View Logs	docker-compose logs -f	tail -f logs/*.log
Restart Backend	docker-compose restart backend	pkill -f uvicorn; ./start_server.sh
Backup	./scripts/backup.sh manual	./scripts/backup.sh manual
Restore	./scripts/restore.sh file.tar.gz	./scripts/restore.sh file.tar.gz
Stop Services	docker-compose down	pkill -f uvicorn; pkill -f "next"

---

Version: 1.0
Last Updated: 2026-04-22
Maintained By: Operations Team
Next Review: 2026-05-22