feat(6): phase 6 plan 02 - operational runbook and documentation
- Created OPERATIONAL_RUNBOOK.md: comprehensive step-by-step procedures for both Docker and Standalone deployment modes covering deployment, daily ops, troubleshooting, backup/restore, disaster recovery, scaling, and updates - Created HEALTH_MONITORING_CHECKLIST.md: daily/weekly/monthly health check procedures with alert thresholds and quick troubleshooting reference - Created DISASTER_RECOVERY_PLAN.md: detailed procedures for 6 failure scenarios (database corruption, hardware failure, data center failure, app crash, disk full, network isolation) with RTO/RPO targets - Created CONFIGURATION_REFERENCE.md: complete documentation of all inventory.env parameters for both deployment modes with common scenarios and troubleshooting - Created EMERGENCY_PROCEDURES.md: quick-reference incident response playbook with 7 critical scenarios, decision tree, escalation path, and printable cheat sheet - Created scripts/backup.sh: automated backup script supporting both Docker and Standalone with integrity verification and retention management - Created scripts/restore.sh: restore script with triple confirmation, safety backups, and validation tests for both deployment modes - Created config/backup-cron.sh: installer for daily/weekly automated backup cron jobs (2 AM daily, 3 AM Sunday) All documentation covers dual-deployment modes with shared configuration files. Documentation is operator-ready with copy-paste commands and clear expected outputs.
This commit is contained in:
192
docs/HEALTH_MONITORING_CHECKLIST.md
Normal file
192
docs/HEALTH_MONITORING_CHECKLIST.md
Normal file
@@ -0,0 +1,192 @@
|
||||
# Health Monitoring Checklist
|
||||
|
||||
Use this checklist for daily/weekly health reviews. Adapt commands for Docker or Standalone mode as needed.
|
||||
|
||||
---
|
||||
|
||||
## Daily Checks (5 minutes)
|
||||
|
||||
Print this section and post near the server or set email reminders.
|
||||
|
||||
- [ ] **All services running**
|
||||
- Docker: `docker-compose ps` (expect: All "Up")
|
||||
- Standalone: `ps aux | grep -E "(uvicorn|next)" | grep -v grep` (expect: 2 processes)
|
||||
|
||||
- [ ] **API responsive**
|
||||
- `curl -w "\nHTTP %{http_code}\n" http://localhost:8000/health`
|
||||
- Expected: 200 OK, response <100ms
|
||||
|
||||
- [ ] **Frontend loads**
|
||||
- `curl -w "\nHTTP %{http_code}\n" http://localhost:3000`
|
||||
- Expected: 200 OK
|
||||
|
||||
- [ ] **Recent errors in logs**
|
||||
- Docker: `docker-compose logs | grep ERROR | tail -5`
|
||||
- Standalone: `tail -20 logs/*.log | grep ERROR`
|
||||
- Action: Investigate any ERROR-level logs
|
||||
|
||||
- [ ] **Database accessible**
|
||||
- Docker: `docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"`
|
||||
- Standalone: `sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"`
|
||||
- Action: If fails, restart backend service
|
||||
|
||||
---
|
||||
|
||||
## Weekly Checks (15 minutes)
|
||||
|
||||
- [ ] **Backup completed**
|
||||
- `ls -lh backups/ | head -1`
|
||||
- Check timestamp is within last 24 hours
|
||||
- Action: Manual backup if needed: `./scripts/backup.sh manual`
|
||||
|
||||
- [ ] **Disk usage within limits**
|
||||
- `du -sh data/ config/ backups/`
|
||||
- Expected: data/ <5GB, backups/ <10GB
|
||||
- Action: If backups >10GB, verify cron retention settings
|
||||
|
||||
- [ ] **Database size reasonable**
|
||||
- `du -h data/inventory.db`
|
||||
- Action: If >1GB, consider optimization (vacuum/index)
|
||||
|
||||
- [ ] **Service resource usage**
|
||||
- Docker: `docker stats --no-stream`
|
||||
- Standalone: `ps aux | grep -E "(uvicorn|next)"`
|
||||
- Expected: Backend <70% CPU, <500MB RAM
|
||||
|
||||
- [ ] **Log files not growing excessively**
|
||||
- `ls -lh logs/ | tail -10`
|
||||
- Action: If any log >100MB, consider rotation
|
||||
|
||||
- [ ] **Check for hung processes**
|
||||
- `ps aux | grep -E "(defunct|defunct)"` (should be empty)
|
||||
- Action: Kill hung processes
|
||||
|
||||
---
|
||||
|
||||
## Monthly Checks (30 minutes)
|
||||
|
||||
- [ ] **Restore from backup test**
|
||||
- On staging environment:
|
||||
```bash
|
||||
./scripts/restore.sh backups/latest.tar.gz --validate
|
||||
```
|
||||
- Confirm zero data loss, all services healthy
|
||||
- Action: If fails, investigate and fix immediately
|
||||
|
||||
- [ ] **Scaling capacity review**
|
||||
- Current: Single instance, 5 concurrent users stable
|
||||
- Actual usage: _____ concurrent users
|
||||
- Action: If approaching 5 users, plan for v3 multi-instance
|
||||
|
||||
- [ ] **Security audit**
|
||||
- [ ] JWT_SECRET_KEY still secure (not exposed in logs)
|
||||
- [ ] LDAP credentials (if used) still valid
|
||||
- [ ] API logs show no unauthorized access attempts
|
||||
- Check: `docker-compose logs backend | grep -i "denied\|failed\|unauthorized" | tail -20`
|
||||
|
||||
- [ ] **Documentation review**
|
||||
- [ ] Runbook matches current deployment
|
||||
- [ ] Troubleshooting section covers recent issues
|
||||
- [ ] Contact info still current
|
||||
|
||||
---
|
||||
|
||||
## Alert Thresholds
|
||||
|
||||
| Metric | Warning | Critical | Action |
|
||||
|--------|---------|----------|--------|
|
||||
| CPU (backend) | >50% | >70% | Restart container, investigate slow queries |
|
||||
| Memory (backend) | >400MB | >600MB | Restart container, check for memory leak |
|
||||
| Disk (backups) | >10GB | >15GB | Delete old backups, increase retention |
|
||||
| API response (p95) | >500ms | >1s | Check slow query logs, restart backend |
|
||||
| Backup age | >36 hours | >48 hours | Manual run needed, check cron |
|
||||
| Database locked | 1 event/week | 5+ events/week | Investigate, may need v3 upgrade |
|
||||
| Error rate | >0.1% | >1% | Investigate logs, restart if needed |
|
||||
|
||||
---
|
||||
|
||||
## Quick Troubleshooting Reference
|
||||
|
||||
**Service down?**
|
||||
→ `docker-compose ps` or `ps aux | grep uvicorn`
|
||||
→ `docker-compose logs SERVICE_NAME` or `tail logs/*.log`
|
||||
→ `docker-compose restart SERVICE_NAME` or `pkill -f uvicorn`
|
||||
|
||||
**Slow responses?**
|
||||
→ `docker stats` or `ps aux | grep uvicorn`
|
||||
→ `docker-compose logs backend | grep "slow"` or check logs
|
||||
→ Restart backend or plan for capacity increase
|
||||
|
||||
**Database locked?**
|
||||
→ Restart backend: `docker-compose restart backend`
|
||||
|
||||
**Out of disk space?**
|
||||
→ `du -sh data/ backups/`
|
||||
→ Clean old backups: `find backups/ -name "*.tar.gz" -mtime +30 -delete`
|
||||
→ Extend volume if needed
|
||||
|
||||
**HTTPS certificate issues?**
|
||||
→ `rm -rf data/caddy_*` (Docker mode)
|
||||
→ `docker-compose restart proxy`
|
||||
→ Wait 30 seconds for new certs to generate
|
||||
|
||||
---
|
||||
|
||||
## Health Check Commands by Deployment Mode
|
||||
|
||||
### Docker Mode
|
||||
|
||||
```bash
|
||||
# Full health check suite
|
||||
echo "=== Services ===" && docker-compose ps
|
||||
echo "=== API Health ===" && curl -s http://localhost:8000/health | jq .
|
||||
echo "=== Database ===" && docker-compose exec backend sqlite3 /app/data/inventory.db "SELECT COUNT(*) FROM items;"
|
||||
echo "=== Resources ===" && docker stats --no-stream | head -5
|
||||
echo "=== Recent Errors ===" && docker-compose logs --tail=20 | grep ERROR
|
||||
```
|
||||
|
||||
### Standalone Mode
|
||||
|
||||
```bash
|
||||
# Full health check suite
|
||||
echo "=== Processes ===" && ps aux | grep -E "(uvicorn|next)" | grep -v grep
|
||||
echo "=== API Health ===" && curl -s http://localhost:8000/health
|
||||
echo "=== Database ===" && sqlite3 data/inventory.db "SELECT COUNT(*) FROM items;"
|
||||
echo "=== Resources ===" && top -bn1 | head -20
|
||||
echo "=== Recent Errors ===" && tail -50 logs/*.log | grep ERROR
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Checklist Template
|
||||
|
||||
Print and use weekly:
|
||||
|
||||
```
|
||||
Week of: ___________
|
||||
|
||||
Daily (✓ = pass, X = fail, note issues):
|
||||
Mon: [ ] Services [ ] API [ ] DB [ ] Errors Notes: _______________
|
||||
Tue: [ ] Services [ ] API [ ] DB [ ] Errors Notes: _______________
|
||||
Wed: [ ] Services [ ] API [ ] DB [ ] Errors Notes: _______________
|
||||
Thu: [ ] Services [ ] API [ ] DB [ ] Errors Notes: _______________
|
||||
Fri: [ ] Services [ ] API [ ] DB [ ] Errors Notes: _______________
|
||||
Sat: [ ] Services [ ] API [ ] DB [ ] Errors Notes: _______________
|
||||
Sun: [ ] Services [ ] API [ ] DB [ ] Errors Notes: _______________
|
||||
|
||||
Weekly Review:
|
||||
- [ ] Backup completed within 24h
|
||||
- [ ] Disk usage acceptable
|
||||
- [ ] DB size reasonable
|
||||
- [ ] Resource usage normal
|
||||
- [ ] No log errors unresolved
|
||||
|
||||
Issues Found: ___________________________________________________
|
||||
Actions Taken: __________________________________________________
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-04-22
|
||||
**Next Review**: 2026-05-22
|
||||
**Maintained By**: Operations Team
|
||||
Reference in New Issue
Block a user