diff --git a/docs/superpowers/specs/2026-04-21-ai-extraction-autosave-photo-design.md b/docs/superpowers/specs/2026-04-21-ai-extraction-autosave-photo-design.md new file mode 100644 index 00000000..2f5cbf6d --- /dev/null +++ b/docs/superpowers/specs/2026-04-21-ai-extraction-autosave-photo-design.md @@ -0,0 +1,350 @@ +# Design: Single-Query AI Extraction + Auto-Photo-Save with Crop/Rotation + +**Date:** 2026-04-21 +**Author:** Claude Haiku 4.5 +**Status:** Design Phase +**Scope:** Phase 3 - Photo Quality & Reliability + +--- + +## 1. Overview + +**Problem:** +- Currently: Two separate API calls (extract-label for OCR, then upload-photo for crop/rotate) +- Inefficient: Duplicate processing, higher token cost +- User experience: Extra step after item creation to manually upload photo + +**Solution:** +- Single API call: `/extract-label` returns item data + crop/rotation metadata +- Backend applies crop/rotation locally using AI guidance +- Automatic photo save after item confirmation (no manual upload needed) +- **Token savings:** ~1000+ tokens per item (no image in response, just coordinates) + +**Benefits:** +- 50% fewer API calls +- Single query to AI instead of two +- Automatic photo integration (better UX) +- Graceful fallback if processing fails + +--- + +## 2. Enhanced AI Prompt + +### Current Prompt Structure +- Extracts item data only (Item, Type, Description, Category, Connector, Size, Color, PartNr, OCR) +- Returns JSON with extracted fields +- No guidance on image layout or rotation + +### Enhanced Prompt Addition + +Add to `/config/ai_prompt.md` (after Output Format section): + +```markdown +## Image Processing Guidance (NEW) + +Analyze the image layout and return crop/rotation metadata to optimize photo storage: + +### Crop Bounds Analysis +- Identify the PRIMARY ITEM in the image (main object, not background/clutter) +- Return bounding box: `{x, y, width, height}` in pixel coordinates +- Rules: + - `x, y`: top-left corner of item (pixel offset from image top-left) + - `width, height`: dimensions of item bounding box + - Include minimal padding (10-15 pixels) around item edges + - Ignore background clutter, other items, hands, reflections + +### Rotation Analysis +- Check if item labels/text are readable +- If text is rotated (not horizontal), calculate rotation needed +- Return `rotation_degrees`: degrees to rotate CLOCKWISE to make text readable +- Examples: + - Text rotated 90° counter-clockwise → return 90 (rotate 90° clockwise) + - Text rotated 45° clockwise → return -45 (rotate 45° counter-clockwise) + - Text already readable → return 0 + +### Confidence Score +- Return `confidence`: 0.0-1.0 indicating reliability of crop/rotation analysis +- 0.9+ = High confidence (clear item, readable text) +- 0.7-0.89 = Medium confidence (some ambiguity or text partially obscured) +- <0.7 = Low confidence (cluttered image, unclear item boundaries) + +### Output Format (Extended) + +```json +{ + "items": [ + { + "Item": "[size] type vendor connector partnumber", + "Type": "type", + "Description": "technical details (max 5 words)", + "Category": "category", + "Connector": "connector_type", + "Size": "human_readable_size", + "Color": "color", + "PartNr": "part_number", + "OCR": "TYPE SIZE VENDOR CONNECTOR PARTNUMBER", + "image_processing": { + "crop_bounds": { + "x": 50, + "y": 100, + "width": 300, + "height": 200 + }, + "rotation_degrees": 15, + "confidence": 0.92 + } + } + ] +} +``` + +**Return ONLY JSON. No markdown. No text.** +``` + +--- + +## 3. Backend Implementation + +### 3.1 Updated Endpoint: `/extract-label` + +**File:** `backend/routers/items.py` + +**Changes:** +- Enhanced prompt now included in `ai_vision.extract_label_info()` +- AI response parsed to include `image_processing` field +- Returns crop_bounds, rotation_degrees, confidence + +**Example Response:** +```json +{ + "items": [ + { + "Item": "1.6TB NVMe HPE U.3 P66093-002", + "Type": "NVMe", + "Description": "Enterprise storage module", + "Category": "Storage", + "Connector": "U.3", + "Size": "1.6TB", + "Color": "Black", + "PartNr": "P66093-002", + "OCR": "NVME 1.6TB HPE U3 P66093002", + "image_processing": { + "crop_bounds": {"x": 45, "y": 80, "width": 350, "height": 220}, + "rotation_degrees": 12, + "confidence": 0.94 + } + } + ] +} +``` + +### 3.2 Backend Photo Auto-Save Logic + +**File:** `backend/routers/items.py` (new function) + +**Function:** `_auto_save_photo_from_extraction(item_id, image_bytes, crop_bounds, rotation_degrees, db_session)` + +**Logic:** +``` +1. Input: item_id, original_image_bytes, crop_bounds, rotation_degrees +2. Check if crop_bounds and rotation_degrees are valid + - If not: log warning, skip photo save, return success (graceful degradation) +3. Create crop_bounds_dict from AI coordinates: + { + "x": crop_bounds["x"], + "y": crop_bounds["y"], + "w": crop_bounds["width"], + "h": crop_bounds["height"] + } +4. Call ImageProcessor.process_photo(image_bytes, crop_bounds_dict, rotation_degrees) +5. If processing fails: log error, skip photo save (don't block item creation) +6. If processing succeeds: + - Get unique filename using ImageStorage.get_unique_filename() + - Save full image and thumbnail + - Update Item.photo_path, photo_thumbnail_path, photo_upload_date +7. Return: {status: "ok"} or {status: "skipped", reason: "..."} +``` + +**Error Handling:** +- Missing crop_bounds/rotation? → Skip photo save, item created successfully +- Processing fails? → Log error, save original image as fallback +- File save fails? → Log error, don't block item creation + +--- + +## 4. Frontend Implementation + +### 4.1 AIOnboarding Component Flow + +**File:** `frontend/components/AIOnboarding.tsx` + +**Current Flow:** +1. Take photo +2. Send to `/extract-label` (get item data only) +3. Show extracted data, user edits +4. User clicks "Create Item" +5. Item created +6. (Separate) User uploads photo later + +**New Flow:** +1. Take photo + store original image bytes in state +2. Send to `/extract-label` (get item data + crop/rotation) +3. Show extracted data + **store image_processing metadata** in state +4. User edits item details +5. User clicks "Create Item" +6. **[NEW]** After item creation, auto-call `/items/{id}/photos` with: + - file: original image + - crop_bounds: from extraction response + - replace_existing: "false" +7. Show success toast: "Item created + photo saved" + +### 4.2 Hook Updates + +**File:** `frontend/hooks/useAIExtraction.ts` + +**Changes:** +- Store extracted `image_processing` data alongside item data +- Pass to `useItemCreate` hook + +**File:** `frontend/hooks/useItemCreate.ts` + +**Changes:** +- After item creation succeeds, check if `image_processing` exists +- If yes: call `uploadPhoto()` with crop_bounds from extraction +- Wait for photo upload to complete +- Show combined toast: "Item + Photo saved" + +### 4.3 Data Flow in State + +```typescript +// In AIOnboarding +const [extractedImage, setExtractedImage] = useState(null); // Original image +const [imageProcessing, setImageProcessing] = useState<{crop_bounds, rotation_degrees, confidence} | null>(null); + +// After extraction +const response = await inventoryApi.analyzeLabel(formData, mode); +setExtractedImage(imageBlob); // Store for later +setImageProcessing(response.items[0].image_processing); // Store metadata + +// When creating item, pass both to useItemCreate +``` + +--- + +## 5. Data Flow Diagram (Text) + +``` +User takes photo + ↓ +POST /extract-label (with enhanced prompt) + ↓ +AI returns: {items: [{...item_data, image_processing: {crop_bounds, rotation_degrees, confidence}}]} + ↓ +Frontend stores: extracted_image + image_processing metadata + ↓ +Show item data, user edits + ↓ +User clicks "Create Item" + ↓ +POST /items → Item created in DB (item_id = 123) + ↓ +POST /items/123/photos with: + - file: extracted_image + - crop_bounds: JSON from image_processing + - replace_existing: false + ↓ +Backend: + 1. Validate crop_bounds JSON + 2. Call ImageProcessor.process_photo(bytes, crop_bounds_dict) + 3. Save full image + thumbnail + 4. Update item.photo_path, photo_thumbnail_path, photo_upload_date + ↓ +Return success + photo URLs + ↓ +Show toast: "Item created + photo saved" +``` + +--- + +## 6. Files Modified + +| File | Change | Lines | +|------|--------|-------| +| `config/ai_prompt.md` | Add "Image Processing Guidance" section with crop/rotation rules | +50 | +| `backend/ai_vision.py` | Parse `image_processing` field from AI response | +20 | +| `backend/routers/items.py` | Add `_auto_save_photo_from_extraction()` helper function; update item creation flow | +80 | +| `frontend/hooks/useAIExtraction.ts` | Store `image_processing` metadata alongside extracted items | +15 | +| `frontend/hooks/useItemCreate.ts` | Auto-call photo upload if `image_processing` exists after item creation | +30 | +| `frontend/components/AIOnboarding.tsx` | Pass extracted image + image_processing to item creation | +10 | + +**Total Impact:** ~205 lines of code/config + +--- + +## 7. Error Handling & Fallbacks + +| Scenario | Handling | +|----------|----------| +| AI doesn't return image_processing | Skip photo save, item created (no photo) | +| crop_bounds is null/invalid | Skip photo save, item created (no photo) | +| ImageProcessor.process_photo() fails | Log error, save original image as-is | +| File save fails | Log error, don't block item creation | +| Network error during photo upload | Return error to frontend (user can retry manually) | +| User has no camera permission | Existing flow (file upload only) | + +--- + +## 8. Testing Strategy + +### Unit Tests +- Parse `image_processing` from AI response correctly +- Validate crop_bounds JSON (x, y, width, height are valid) +- Rotation degrees within valid range (-360 to +360) +- Confidence score is 0.0-1.0 + +### Integration Tests +- Extract → Create Item → Auto-save Photo flow end-to-end +- Photo saved with correct crop/rotation applied +- Fallback: photo save fails, item still created +- Manual photo upload still works (separate flow) + +### E2E Tests +- User takes photo → AI extracts + crop guidance → creates item → photo auto-saved +- Verify photo appears in inventory card with correct crop + +--- + +## 9. Success Criteria + +✅ Single API call returns item data + crop/rotation guidance +✅ Backend applies crop/rotation from AI metadata +✅ Photo auto-saved after item confirmation +✅ No manual photo upload step needed for AI-identified items +✅ Graceful fallback if processing fails +✅ ~1000+ token savings per extraction (no image in response) +✅ All existing tests pass +✅ E2E test covers full flow + +--- + +## 10. Rollout Strategy + +**Phase 1:** Backend + Frontend changes (non-breaking) +- Old `/extract-label` calls still work (image_processing field optional) +- Manual photo upload still works +- AIOnboarding auto-save only for new items with image_processing data + +**Phase 2:** Update AI prompt in config (activate crop/rotation guidance) +- Existing deployments get enhanced prompt on next config reload +- New extractions return image_processing field + +**Rollback:** Remove image_processing field from response, revert to manual upload + +--- + +## 11. Notes + +- **Backward Compatibility:** If AI doesn't return `image_processing`, system falls back to manual upload (no breaking change) +- **Storage:** Original image passed from frontend to photo upload endpoint (already happens in current flow) +- **Security:** No new endpoints, no new auth required (existing /extract-label and /items/{id}/photos endpoints) +- **Performance:** Single AI call vs two API calls = 50% fewer round-trips