# Health Monitoring Checklist

Use this checklist for daily/weekly health reviews. Adapt commands for Docker or Standalone mode as needed.

---

## Daily Checks (5 minutes)

Print this section and post near the server or set email reminders.

- [ ] **All services running**
  - Docker: `docker-compose ps` (expect: All "Up")
  - Standalone: `ps aux | grep -E "(uvicorn|next)" | grep -v grep` (expect: 2 processes)

- [ ] **API responsive**
  - `curl -w "\nHTTP %{http_code}\n" http://localhost:8000/health`
  - Expected: 200 OK, response <100ms

- [ ] **Frontend loads**
  - `curl -w "\nHTTP %{http_code}\n" http://localhost:3000`
  - Expected: 200 OK

- [ ] **Recent errors in logs**
  - Docker: `docker-compose logs | grep ERROR | tail -5`
  - Standalone: `tail -20 logs/*.log | grep ERROR`
  - Action: Investigate any ERROR-level logs

- [ ] **Database accessible**
  - Docker: `docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"`
  - Standalone: `sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"`
  - Action: If fails, restart backend service

---

## Weekly Checks (15 minutes)

- [ ] **Backup completed**
  - `ls -lh backups/ | head -1`
  - Check timestamp is within last 24 hours
  - Action: Manual backup if needed: `./scripts/backup.sh manual`

- [ ] **Disk usage within limits**
  - `du -sh data/ config/ backups/`
  - Expected: data/ <5GB, backups/ <10GB
  - Action: If backups >10GB, verify cron retention settings

- [ ] **Database size reasonable**
  - `du -h data/inventory.db`
  - Action: If >1GB, consider optimization (vacuum/index)

- [ ] **Service resource usage**
  - Docker: `docker stats --no-stream`
  - Standalone: `ps aux | grep -E "(uvicorn|next)"`
  - Expected: Backend <70% CPU, <500MB RAM

- [ ] **Log files not growing excessively**
  - `ls -lh logs/ | tail -10`
  - Action: If any log >100MB, consider rotation

- [ ] **Check for hung processes**
  - `ps aux | grep -E "(defunct|defunct)"` (should be empty)
  - Action: Kill hung processes

---

## Monthly Checks (30 minutes)

- [ ] **Restore from backup test**
  - On staging environment:
    ```bash
    ./scripts/restore.sh backups/latest.tar.gz --validate
    ```
  - Confirm zero data loss, all services healthy
  - Action: If fails, investigate and fix immediately

- [ ] **Scaling capacity review**
  - Current: Single instance, 5 concurrent users stable
  - Actual usage: _____ concurrent users
  - Action: If approaching 5 users, plan for v3 multi-instance

- [ ] **Security audit**
  - [ ] JWT_SECRET_KEY still secure (not exposed in logs)
  - [ ] LDAP credentials (if used) still valid
  - [ ] API logs show no unauthorized access attempts
  - Check: `docker-compose logs backend | grep -i "denied\|failed\|unauthorized" | tail -20`

- [ ] **Documentation review**
  - [ ] Runbook matches current deployment
  - [ ] Troubleshooting section covers recent issues
  - [ ] Contact info still current

---

## Alert Thresholds

| Metric | Warning | Critical | Action |
|--------|---------|----------|--------|
| CPU (backend) | >50% | >70% | Restart container, investigate slow queries |
| Memory (backend) | >400MB | >600MB | Restart container, check for memory leak |
| Disk (backups) | >10GB | >15GB | Delete old backups, increase retention |
| API response (p95) | >500ms | >1s | Check slow query logs, restart backend |
| Backup age | >36 hours | >48 hours | Manual run needed, check cron |
| Database locked | 1 event/week | 5+ events/week | Investigate, may need v3 upgrade |
| Error rate | >0.1% | >1% | Investigate logs, restart if needed |

---

## Quick Troubleshooting Reference

**Service down?**
→ `docker-compose ps` or `ps aux | grep uvicorn`
→ `docker-compose logs SERVICE_NAME` or `tail logs/*.log`
→ `docker-compose restart SERVICE_NAME` or `pkill -f uvicorn`

**Slow responses?**
→ `docker stats` or `ps aux | grep uvicorn`
→ `docker-compose logs backend | grep "slow"` or check logs
→ Restart backend or plan for capacity increase

**Database locked?**
→ Restart backend: `docker-compose restart backend`

**Out of disk space?**
→ `du -sh data/ backups/`
→ Clean old backups: `find backups/ -name "*.tar.gz" -mtime +30 -delete`
→ Extend volume if needed

**HTTPS certificate issues?**
→ `rm -rf data/caddy_*` (Docker mode)
→ `docker-compose restart proxy`
→ Wait 30 seconds for new certs to generate

---

## Health Check Commands by Deployment Mode

### Docker Mode

```bash
# Full health check suite
echo "=== Services ===" && docker-compose ps
echo "=== API Health ===" && curl -s http://localhost:8000/health | jq .
echo "=== Database ===" && docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;" 
echo "=== Resources ===" && docker stats --no-stream | head -5
echo "=== Recent Errors ===" && docker-compose logs --tail=20 | grep ERROR
```

### Standalone Mode

```bash
# Full health check suite
echo "=== Processes ===" && ps aux | grep -E "(uvicorn|next)" | grep -v grep
echo "=== API Health ===" && curl -s http://localhost:8000/health
echo "=== Database ===" && sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"
echo "=== Resources ===" && top -bn1 | head -20
echo "=== Recent Errors ===" && tail -50 logs/*.log | grep ERROR
```

---

## Monitoring Checklist Template

Print and use weekly:

```
Week of: ___________

Daily (✓ = pass, X = fail, note issues):
Mon: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Tue: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Wed: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Thu: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Fri: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Sat: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________
Sun: [ ] Services  [ ] API  [ ] DB  [ ] Errors    Notes: _______________

Weekly Review:
- [ ] Backup completed within 24h
- [ ] Disk usage acceptable
- [ ] DB size reasonable
- [ ] Resource usage normal
- [ ] No log errors unresolved

Issues Found: ___________________________________________________
Actions Taken: __________________________________________________
```

---

**Last Updated**: 2026-04-22  
**Next Review**: 2026-05-22  
**Maintained By**: Operations Team