tfm_ainventory/docs/HEALTH_MONITORING_CHECKLIST.md

Health Monitoring Checklist

Use this checklist for daily/weekly health reviews. Adapt commands for Docker or Standalone mode as needed.

---

Daily Checks (5 minutes)

Print this section and post near the server or set email reminders.

• All services running

  • Docker: docker-compose ps (expect: All "Up")
  • Standalone: ps aux | grep -E "(uvicorn|next)" | grep -v grep (expect: 2 processes)

• API responsive

  • curl -w "\nHTTP %{http_code}\n" http://localhost:8000/health
  • Expected: 200 OK, response <100ms

• Frontend loads

  • curl -w "\nHTTP %{http_code}\n" http://localhost:3000
  • Expected: 200 OK

• Recent errors in logs

  • Docker: docker-compose logs | grep ERROR | tail -5
  • Standalone: tail -20 logs/*.log | grep ERROR
  • Action: Investigate any ERROR-level logs

• Database accessible

  • Docker: docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"
  • Standalone: sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"
  • Action: If fails, restart backend service

---

Weekly Checks (15 minutes)

• Backup completed

  • ls -lh backups/ | head -1
  • Check timestamp is within last 24 hours
  • Action: Manual backup if needed: ./scripts/backup.sh manual

• Disk usage within limits

  • du -sh data/ config/ backups/
  • Expected: data/ <5GB, backups/ <10GB
  • Action: If backups >10GB, verify cron retention settings

• Database size reasonable

  • du -h data/inventory.db
  • Action: If >1GB, consider optimization (vacuum/index)

• Service resource usage

  • Docker: docker stats --no-stream
  • Standalone: ps aux | grep -E "(uvicorn|next)"
  • Expected: Backend <70% CPU, <500MB RAM

• Log files not growing excessively

  • ls -lh logs/ | tail -10
  • Action: If any log >100MB, consider rotation

• Check for hung processes

  • ps aux | grep -E "(defunct|defunct)" (should be empty)
  • Action: Kill hung processes

---

Monthly Checks (30 minutes)

• Restore from backup test

  • On staging environment:

    ./scripts/restore.sh backups/latest.tar.gz --validate
    

  • Confirm zero data loss, all services healthy
  • Action: If fails, investigate and fix immediately

• Scaling capacity review

  • Current: Single instance, 5 concurrent users stable
  • Actual usage: _____ concurrent users
  • Action: If approaching 5 users, plan for v3 multi-instance

• Security audit

  • JWT_SECRET_KEY still secure (not exposed in logs)
  • LDAP credentials (if used) still valid
  • API logs show no unauthorized access attempts
  • Check: docker-compose logs backend | grep -i "denied\|failed\|unauthorized" | tail -20

• Documentation review

  • Runbook matches current deployment
  • Troubleshooting section covers recent issues
  • Contact info still current

---

Alert Thresholds

Metric	Warning	Critical	Action
CPU (backend)	>50%	>70%	Restart container, investigate slow queries
Memory (backend)	>400MB	>600MB	Restart container, check for memory leak
Disk (backups)	>10GB	>15GB	Delete old backups, increase retention
API response (p95)	>500ms	>1s	Check slow query logs, restart backend
Backup age	>36 hours	>48 hours	Manual run needed, check cron
Database locked	1 event/week	5+ events/week	Investigate, may need v3 upgrade
Error rate	>0.1%	>1%	Investigate logs, restart if needed

---

Quick Troubleshooting Reference

Service down? → docker-compose ps or ps aux | grep uvicorn → docker-compose logs SERVICE_NAME or tail logs/*.log → docker-compose restart SERVICE_NAME or pkill -f uvicorn

Slow responses? → docker stats or ps aux | grep uvicorn → docker-compose logs backend | grep "slow" or check logs → Restart backend or plan for capacity increase

Database locked? → Restart backend: docker-compose restart backend

Out of disk space? → du -sh data/ backups/ → Clean old backups: find backups/ -name "*.tar.gz" -mtime +30 -delete → Extend volume if needed

HTTPS certificate issues? → rm -rf data/caddy_* (Docker mode) → docker-compose restart proxy → Wait 30 seconds for new certs to generate

---

Health Check Commands by Deployment Mode

Docker Mode

# Full health check suite
echo "=== Services ===" && docker-compose ps
echo "=== API Health ===" && curl -s http://localhost:8000/health | jq .
echo "=== Database ===" && docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;" 
echo "=== Resources ===" && docker stats --no-stream | head -5
echo "=== Recent Errors ===" && docker-compose logs --tail=20 | grep ERROR

Standalone Mode

# Full health check suite
echo "=== Processes ===" && ps aux | grep -E "(uvicorn|next)" | grep -v grep
echo "=== API Health ===" && curl -s http://localhost:8000/health
echo "=== Database ===" && sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"
echo "=== Resources ===" && top -bn1 | head -20
echo "=== Recent Errors ===" && tail -50 logs/*.log | grep ERROR

---

Monitoring Checklist Template

Print and use weekly:

Week of: ___________

Daily (✓ = pass, X = fail, note issues):
Mon: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Tue: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Wed: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Thu: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Fri: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Sat: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Sun: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________

Weekly Review:
- [ ] Backup completed within 24h
- [ ] Disk usage acceptable
- [ ] DB size reasonable
- [ ] Resource usage normal
- [ ] No log errors unresolved

Issues Found: ___________________________________________________
Actions Taken: __________________________________________________

---

Last Updated: 2026-04-22
Next Review: 2026-05-22
Maintained By: Operations Team