docs: add phase 3 E2E tests design spec (Playwright, modular workflows, <30min parallel)
This commit is contained in:
425
docs/superpowers/specs/2026-04-19-phase-3-e2e-tests-design.md
Normal file
425
docs/superpowers/specs/2026-04-19-phase-3-e2e-tests-design.md
Normal file
@@ -0,0 +1,425 @@
|
||||
# Phase 3 Design: Playwright E2E Tests (Modular Workflows)
|
||||
|
||||
**Date:** 2026-04-19
|
||||
**Status:** Design Approved
|
||||
**Target Runtime:** <30 minutes (parallel execution)
|
||||
**Test Scope:** 5 critical user workflows + error handling
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
Phase 3 extends Phase 2 (284 unit tests) with end-to-end browser automation tests using Playwright. Each of 5 critical workflows runs in its own isolated Docker environment with a dedicated database, LDAP server, and app instance. Tests run in parallel, completing within <30 minutes.
|
||||
|
||||
**Workflows:**
|
||||
1. Login (LDAP + local authentication)
|
||||
2. Scan → Adjust Stock (barcode matching)
|
||||
3. AI Extraction (new item onboarding)
|
||||
4. Admin Settings (configuration & user management)
|
||||
5. Offline Sync (queue → sync idempotency)
|
||||
|
||||
---
|
||||
|
||||
## 2. Test Architecture
|
||||
|
||||
### 2.1 Directory Structure
|
||||
|
||||
```
|
||||
frontend/e2e/
|
||||
├── workflows/
|
||||
│ ├── 1-login.spec.ts (LDAP + local auth flows)
|
||||
│ ├── 2-scan-adjust.spec.ts (barcode scan, stock adjustment)
|
||||
│ ├── 3-ai-extraction.spec.ts (photo → AI → validation)
|
||||
│ ├── 4-admin-settings.spec.ts (admin dashboard, config changes)
|
||||
│ └── 5-offline-sync.spec.ts (offline ops → sync)
|
||||
├── fixtures/
|
||||
│ ├── db.ts (SQLite seeding, cleanup, per-workflow)
|
||||
│ ├── ldap.ts (OpenLDAP container setup, test users)
|
||||
│ ├── auth.ts (login helpers, session management)
|
||||
│ └── test-data.ts (seed definitions, factories)
|
||||
├── utils/
|
||||
│ ├── assertions.ts (custom Playwright matchers)
|
||||
│ ├── docker.ts (container orchestration, lifecycle)
|
||||
│ └── helpers.ts (navigation, wait conditions)
|
||||
├── docker-compose.e2e.yml (shared services template)
|
||||
├── playwright.config.ts (Playwright configuration)
|
||||
└── README.md (setup & execution guide)
|
||||
```
|
||||
|
||||
### 2.2 Per-Workflow Isolation
|
||||
|
||||
Each workflow runs independently:
|
||||
|
||||
| Workflow | Backend Port | Frontend Port | Database | LDAP | AI Mock |
|
||||
|----------|--------------|---------------|----------|------|---------|
|
||||
| 1-login | 8906 | 8907 | Fresh | Yes | N/A |
|
||||
| 2-scan-adjust | 8916 | 8917 | Seeded (10 items) | No | N/A |
|
||||
| 3-ai-extraction | 8926 | 8927 | Fresh | No | Gemini/Claude mocked |
|
||||
| 4-admin-settings | 8936 | 8937 | Seeded (users, categories) | Yes | Mocked |
|
||||
| 5-offline-sync | 8946 | 8947 | Fresh | No | N/A |
|
||||
|
||||
**Benefits:**
|
||||
- No port conflicts (workflows run in parallel)
|
||||
- Failed workflow doesn't affect others
|
||||
- Per-workflow database cleanup (no state leakage)
|
||||
- Independent LDAP setup for auth workflows
|
||||
|
||||
---
|
||||
|
||||
## 3. Workflow Test Scenarios
|
||||
|
||||
### 3.1 Workflow 1: Login (LDAP + Local)
|
||||
|
||||
**Setup:** LDAP container with test users + app + empty database
|
||||
|
||||
**Scenarios:**
|
||||
1. ✅ LDAP user login (valid credentials → dashboard)
|
||||
2. ✅ Local user login (password → dashboard)
|
||||
3. ✅ Invalid LDAP credentials → error message
|
||||
4. ✅ Invalid local password → error message
|
||||
5. ✅ Missing username/password → validation error
|
||||
6. ✅ Session expiry (token timeout) → redirect to login
|
||||
7. ✅ Logout (clear session) → login screen
|
||||
8. ✅ Concurrent login attempts → proper queueing
|
||||
|
||||
**Error Cases:**
|
||||
- LDAP server down → fallback to local auth
|
||||
- Network timeout → retry with backoff
|
||||
- Invalid token format → re-authenticate
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Workflow 2: Scan → Adjust Stock
|
||||
|
||||
**Setup:** App + seeded database (10 items with barcodes)
|
||||
|
||||
**Scenarios:**
|
||||
1. ✅ Scan valid barcode → match existing item → open adjustment UI
|
||||
2. ✅ Adjust quantity (+5) → confirm → audit log updated
|
||||
3. ✅ Scan unknown barcode → create new item flow
|
||||
4. ✅ Multiple consecutive scans (5+) → batch operations queue
|
||||
5. ✅ Scan while offline → queue operation → sync on reconnect
|
||||
6. ✅ Barcode not found → OCR fallback search
|
||||
7. ✅ Box label scan → multi-item selection UI
|
||||
8. ✅ Concurrent scans → no race conditions
|
||||
|
||||
**Error Cases:**
|
||||
- Barcode decode failure → retry
|
||||
- Network timeout during save → offline queue
|
||||
- Inventory constraint violation (negative qty) → validation error
|
||||
- Concurrent quantity updates → last-write-wins with audit
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Workflow 3: AI Extraction (New Item Onboarding)
|
||||
|
||||
**Setup:** App + empty database + mocked Gemini/Claude APIs
|
||||
|
||||
**Scenarios:**
|
||||
1. ✅ Capture photo → send to AI → receive extraction
|
||||
2. ✅ AI response (name, part number, category) → validation UI
|
||||
3. ✅ Confirm extracted data → save item
|
||||
4. ✅ Reject extraction → manual entry form
|
||||
5. ✅ AI extraction with multiple items in photo
|
||||
6. ✅ Box discovery mode (AI focuses on container labels)
|
||||
7. ✅ AI timeout → retry with exponential backoff
|
||||
8. ✅ Network failure during extraction → offline queue
|
||||
|
||||
**Error Cases:**
|
||||
- Image validation (blur, size, format) → error message
|
||||
- Invalid EXIF data → degrade gracefully
|
||||
- AI service timeout (>10s) → user can retry or enter manually
|
||||
- Malformed AI response → fallback to manual entry
|
||||
- Concurrent extraction requests → queue + process sequentially
|
||||
|
||||
---
|
||||
|
||||
### 3.4 Workflow 4: Admin Settings
|
||||
|
||||
**Setup:** App + seeded database (5 test users, 8 categories) + LDAP
|
||||
|
||||
**Scenarios:**
|
||||
1. ✅ Navigate to Admin Dashboard
|
||||
2. ✅ Identity Manager: list users (LDAP + local)
|
||||
3. ✅ Create new local user → email validation
|
||||
4. ✅ Delete user → confirmation modal → audit log
|
||||
5. ✅ AI Manager: switch provider (Gemini → Claude)
|
||||
6. ✅ Update API key → test connection → success/failure
|
||||
7. ✅ LDAP Manager: update server settings → test connection
|
||||
8. ✅ Database Manager: view backup status → trigger backup
|
||||
9. ✅ Category Manager: add/delete categories
|
||||
10. ✅ Configuration saved → persists across sessions
|
||||
|
||||
**Error Cases:**
|
||||
- Invalid API key → error toast, no save
|
||||
- LDAP connection timeout → error state, keep previous config
|
||||
- Concurrent config updates → optimistic UI + server validation
|
||||
- Missing required fields → inline validation
|
||||
- Database backup failure → error state, rollback
|
||||
|
||||
---
|
||||
|
||||
### 3.5 Workflow 5: Offline Sync
|
||||
|
||||
**Setup:** App + empty database + simulated offline mode
|
||||
|
||||
**Scenarios:**
|
||||
1. ✅ Perform operation (scan, create item) → offline detection
|
||||
2. ✅ Queue 5+ operations while offline
|
||||
3. ✅ Go online → automatic sync batch to server
|
||||
4. ✅ UUID idempotency: sync same batch twice → no duplicates
|
||||
5. ✅ Partial sync failure → retry remaining items
|
||||
6. ✅ Sync with network timeout → exponential backoff
|
||||
7. ✅ Concurrent updates (offline + online) → conflict resolution
|
||||
8. ✅ Local state persists (IndexedDB) → reload page → continues sync
|
||||
|
||||
**Error Cases:**
|
||||
- Sync failure mid-batch → remaining items queued
|
||||
- Server rejects UUID → log error, mark item as failed
|
||||
- IndexedDB quota exceeded → error toast
|
||||
- Corrupted queue entry → skip + continue
|
||||
- Server version mismatch (audit schema) → graceful degradation
|
||||
|
||||
---
|
||||
|
||||
## 4. Error Handling & Resilience
|
||||
|
||||
### 4.1 Network Failures
|
||||
|
||||
**Timeout Handling:**
|
||||
- API call timeout > 10s → retry 2x with exponential backoff (1s, 2s)
|
||||
- Container startup timeout > 30s → fail fast, report health check failure
|
||||
- Page load > 15s → timeout assertion
|
||||
|
||||
**Connection Loss:**
|
||||
- Offline detection: monitor navigator.onLine + failed API call
|
||||
- Offline queue: IndexedDB stores operations with UUID + timestamp
|
||||
- Sync on reconnect: automatic batch send, retry failed items
|
||||
|
||||
### 4.2 Concurrent Operations
|
||||
|
||||
**Race Condition Prevention:**
|
||||
- Scanning: queue concurrent scans, process sequentially
|
||||
- Stock adjustment: last-write-wins with server validation
|
||||
- Config updates: optimistic UI, server validation, rollback on fail
|
||||
- AI extraction: single extraction per session (prevent duplicate calls)
|
||||
|
||||
### 4.3 Invalid Input Handling
|
||||
|
||||
- Image validation (size, format, blur) → inline error
|
||||
- Missing required fields → form validation error
|
||||
- Invalid barcode → OCR fallback + manual entry
|
||||
- Malformed AI response → user can retry or enter manually
|
||||
|
||||
---
|
||||
|
||||
## 5. Docker & Infrastructure
|
||||
|
||||
### 5.1 Docker Compose Setup
|
||||
|
||||
**Base Configuration (`docker-compose.e2e.yml`):**
|
||||
```yaml
|
||||
services:
|
||||
# App backend
|
||||
backend:
|
||||
image: ainventory-backend:test
|
||||
ports:
|
||||
- "${BACKEND_PORT}:8906"
|
||||
environment:
|
||||
DATABASE_URL: sqlite:///test-${WORKFLOW_ID}.db
|
||||
LDAP_ENABLED: "${LDAP_ENABLED}"
|
||||
AI_PROVIDER: "${AI_PROVIDER}"
|
||||
GEMINI_API_KEY: "test-key"
|
||||
CLAUDE_API_KEY: "test-key"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8906/health"]
|
||||
interval: 2s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
||||
# Frontend dev server
|
||||
frontend:
|
||||
image: node:20
|
||||
working_dir: /app
|
||||
ports:
|
||||
- "${FRONTEND_PORT}:8907"
|
||||
environment:
|
||||
NEXT_PUBLIC_API_URL: "http://localhost:${BACKEND_PORT}"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:3000"]
|
||||
interval: 2s
|
||||
retries: 10
|
||||
|
||||
# OpenLDAP (for auth workflows)
|
||||
ldap:
|
||||
image: osixia/openldap:latest
|
||||
ports:
|
||||
- "${LDAP_PORT}:389"
|
||||
environment:
|
||||
LDAP_ORGANISATION: "aInventory"
|
||||
LDAP_BASE_DN: "dc=ainventory,dc=local"
|
||||
```
|
||||
|
||||
### 5.2 Container Lifecycle
|
||||
|
||||
**Per-Workflow:**
|
||||
1. **Setup Phase (~15-20s)**
|
||||
- Start Docker Compose for workflow
|
||||
- Wait for health checks (backend, frontend, LDAP if needed)
|
||||
- Seed database (SQL migrations)
|
||||
- Pre-populate LDAP users (if needed)
|
||||
|
||||
2. **Test Phase (~3-5 min)**
|
||||
- Playwright runs test scenarios
|
||||
- Browser automation against live app
|
||||
- Real API calls to backend
|
||||
|
||||
3. **Teardown Phase (~5-10s)**
|
||||
- Stop all containers
|
||||
- Clean database volume
|
||||
- Collect logs for debugging
|
||||
|
||||
---
|
||||
|
||||
## 6. Test Configuration
|
||||
|
||||
### 6.1 Playwright Config
|
||||
|
||||
```typescript
|
||||
// playwright.config.ts
|
||||
export default defineConfig({
|
||||
testDir: './e2e/workflows',
|
||||
fullyParallel: true,
|
||||
workers: 5, // Run 5 workflows in parallel
|
||||
timeout: 30000, // 30s per test
|
||||
expect: { timeout: 5000 },
|
||||
webServer: [], // No webServer (Docker manages this)
|
||||
use: {
|
||||
baseURL: 'http://localhost', // Dynamic per workflow
|
||||
trace: 'on-first-retry',
|
||||
screenshot: 'only-on-failure',
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### 6.2 Environment Setup
|
||||
|
||||
**Env Variables per Workflow:**
|
||||
```bash
|
||||
# .env.e2e.workflow-1
|
||||
BACKEND_PORT=8906
|
||||
FRONTEND_PORT=8907
|
||||
LDAP_ENABLED=true
|
||||
LDAP_PORT=3389
|
||||
AI_PROVIDER=gemini
|
||||
|
||||
# .env.e2e.workflow-2
|
||||
BACKEND_PORT=8916
|
||||
FRONTEND_PORT=8917
|
||||
LDAP_ENABLED=false
|
||||
AI_PROVIDER=gemini
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Test Execution & CI/CD
|
||||
|
||||
### 7.1 Local Execution
|
||||
|
||||
```bash
|
||||
# Run all workflows in parallel
|
||||
npm run e2e
|
||||
|
||||
# Run specific workflow
|
||||
npm run e2e -- workflows/1-login.spec.ts
|
||||
|
||||
# Debug mode (headed browser)
|
||||
npm run e2e:debug
|
||||
```
|
||||
|
||||
### 7.2 Expected Runtime
|
||||
|
||||
- **Per Workflow:** 3-5 minutes
|
||||
- **Sequential Total:** 15-25 minutes
|
||||
- **Parallel Total:** 8-10 minutes (5 workers)
|
||||
- **Target:** <30 minutes ✅
|
||||
|
||||
### 7.3 CI/CD Integration
|
||||
|
||||
```bash
|
||||
# GitHub Actions / Local CI
|
||||
npm run build
|
||||
npm run e2e -- --reporter=html
|
||||
# Report: playwright-report/index.html
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Success Criteria
|
||||
|
||||
✅ All 5 workflows tested
|
||||
✅ 40+ test cases across workflows
|
||||
✅ Error scenarios included
|
||||
✅ Parallel execution <30 min
|
||||
✅ Zero flaky tests (3x runs stable)
|
||||
✅ Comprehensive error handling
|
||||
✅ Docker isolation working
|
||||
✅ Database cleanup per workflow
|
||||
✅ HTML report generated
|
||||
|
||||
---
|
||||
|
||||
## 9. Scope & Constraints
|
||||
|
||||
**In Scope:**
|
||||
- Happy path workflows
|
||||
- Critical error scenarios (network, auth, validation)
|
||||
- Concurrent operation handling
|
||||
- Offline → online sync
|
||||
- Docker-based isolation
|
||||
|
||||
**Out of Scope:**
|
||||
- Performance benchmarking
|
||||
- Load testing
|
||||
- Mobile-specific gestures (covered by Vitest unit tests)
|
||||
- Visual regression testing
|
||||
- Accessibility audits (covered by Phase 2)
|
||||
|
||||
---
|
||||
|
||||
## 10. Dependencies & Prerequisites
|
||||
|
||||
**Required:**
|
||||
- Docker & Docker Compose
|
||||
- Node.js 20+
|
||||
- Playwright (`@playwright/test`)
|
||||
- Python 3.12+ (backend venv)
|
||||
|
||||
**Optional:**
|
||||
- `docker-compose` plugin
|
||||
- `curl` (for health checks)
|
||||
|
||||
---
|
||||
|
||||
## 11. Risk Mitigation
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Docker startup slow | Health checks + parallel workers |
|
||||
| Flaky network tests | Retry logic + exponential backoff |
|
||||
| Port conflicts | Offset ports per workflow (8906, 8916, 8926, etc.) |
|
||||
| Database state leakage | Fresh DB per workflow, cleanup after |
|
||||
| LDAP timeout | Fallback to local auth, skip LDAP tests if unavailable |
|
||||
| Concurrent AI calls | Queue extraction requests, single-at-a-time processing |
|
||||
|
||||
---
|
||||
|
||||
## 12. Next Steps
|
||||
|
||||
1. ✅ Design approved
|
||||
2. → Create implementation plan (writing-plans skill)
|
||||
3. → Install Playwright, set up docker-compose.e2e.yml
|
||||
4. → Build test fixtures (db, ldap, auth)
|
||||
5. → Implement 5 workflow test files
|
||||
6. → Verify parallel execution <30 min
|
||||
7. → Commit & tag `phase-3-complete`
|
||||
Reference in New Issue
Block a user