docs: design AI extraction + auto-photo-save with crop/rotation guidance
This commit is contained in:
@@ -0,0 +1,350 @@
|
||||
# Design: Single-Query AI Extraction + Auto-Photo-Save with Crop/Rotation
|
||||
|
||||
**Date:** 2026-04-21
|
||||
**Author:** Claude Haiku 4.5
|
||||
**Status:** Design Phase
|
||||
**Scope:** Phase 3 - Photo Quality & Reliability
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview
|
||||
|
||||
**Problem:**
|
||||
- Currently: Two separate API calls (extract-label for OCR, then upload-photo for crop/rotate)
|
||||
- Inefficient: Duplicate processing, higher token cost
|
||||
- User experience: Extra step after item creation to manually upload photo
|
||||
|
||||
**Solution:**
|
||||
- Single API call: `/extract-label` returns item data + crop/rotation metadata
|
||||
- Backend applies crop/rotation locally using AI guidance
|
||||
- Automatic photo save after item confirmation (no manual upload needed)
|
||||
- **Token savings:** ~1000+ tokens per item (no image in response, just coordinates)
|
||||
|
||||
**Benefits:**
|
||||
- 50% fewer API calls
|
||||
- Single query to AI instead of two
|
||||
- Automatic photo integration (better UX)
|
||||
- Graceful fallback if processing fails
|
||||
|
||||
---
|
||||
|
||||
## 2. Enhanced AI Prompt
|
||||
|
||||
### Current Prompt Structure
|
||||
- Extracts item data only (Item, Type, Description, Category, Connector, Size, Color, PartNr, OCR)
|
||||
- Returns JSON with extracted fields
|
||||
- No guidance on image layout or rotation
|
||||
|
||||
### Enhanced Prompt Addition
|
||||
|
||||
Add to `/config/ai_prompt.md` (after Output Format section):
|
||||
|
||||
```markdown
|
||||
## Image Processing Guidance (NEW)
|
||||
|
||||
Analyze the image layout and return crop/rotation metadata to optimize photo storage:
|
||||
|
||||
### Crop Bounds Analysis
|
||||
- Identify the PRIMARY ITEM in the image (main object, not background/clutter)
|
||||
- Return bounding box: `{x, y, width, height}` in pixel coordinates
|
||||
- Rules:
|
||||
- `x, y`: top-left corner of item (pixel offset from image top-left)
|
||||
- `width, height`: dimensions of item bounding box
|
||||
- Include minimal padding (10-15 pixels) around item edges
|
||||
- Ignore background clutter, other items, hands, reflections
|
||||
|
||||
### Rotation Analysis
|
||||
- Check if item labels/text are readable
|
||||
- If text is rotated (not horizontal), calculate rotation needed
|
||||
- Return `rotation_degrees`: degrees to rotate CLOCKWISE to make text readable
|
||||
- Examples:
|
||||
- Text rotated 90° counter-clockwise → return 90 (rotate 90° clockwise)
|
||||
- Text rotated 45° clockwise → return -45 (rotate 45° counter-clockwise)
|
||||
- Text already readable → return 0
|
||||
|
||||
### Confidence Score
|
||||
- Return `confidence`: 0.0-1.0 indicating reliability of crop/rotation analysis
|
||||
- 0.9+ = High confidence (clear item, readable text)
|
||||
- 0.7-0.89 = Medium confidence (some ambiguity or text partially obscured)
|
||||
- <0.7 = Low confidence (cluttered image, unclear item boundaries)
|
||||
|
||||
### Output Format (Extended)
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"Item": "[size] type vendor connector partnumber",
|
||||
"Type": "type",
|
||||
"Description": "technical details (max 5 words)",
|
||||
"Category": "category",
|
||||
"Connector": "connector_type",
|
||||
"Size": "human_readable_size",
|
||||
"Color": "color",
|
||||
"PartNr": "part_number",
|
||||
"OCR": "TYPE SIZE VENDOR CONNECTOR PARTNUMBER",
|
||||
"image_processing": {
|
||||
"crop_bounds": {
|
||||
"x": 50,
|
||||
"y": 100,
|
||||
"width": 300,
|
||||
"height": 200
|
||||
},
|
||||
"rotation_degrees": 15,
|
||||
"confidence": 0.92
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Return ONLY JSON. No markdown. No text.**
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Backend Implementation
|
||||
|
||||
### 3.1 Updated Endpoint: `/extract-label`
|
||||
|
||||
**File:** `backend/routers/items.py`
|
||||
|
||||
**Changes:**
|
||||
- Enhanced prompt now included in `ai_vision.extract_label_info()`
|
||||
- AI response parsed to include `image_processing` field
|
||||
- Returns crop_bounds, rotation_degrees, confidence
|
||||
|
||||
**Example Response:**
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"Item": "1.6TB NVMe HPE U.3 P66093-002",
|
||||
"Type": "NVMe",
|
||||
"Description": "Enterprise storage module",
|
||||
"Category": "Storage",
|
||||
"Connector": "U.3",
|
||||
"Size": "1.6TB",
|
||||
"Color": "Black",
|
||||
"PartNr": "P66093-002",
|
||||
"OCR": "NVME 1.6TB HPE U3 P66093002",
|
||||
"image_processing": {
|
||||
"crop_bounds": {"x": 45, "y": 80, "width": 350, "height": 220},
|
||||
"rotation_degrees": 12,
|
||||
"confidence": 0.94
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 Backend Photo Auto-Save Logic
|
||||
|
||||
**File:** `backend/routers/items.py` (new function)
|
||||
|
||||
**Function:** `_auto_save_photo_from_extraction(item_id, image_bytes, crop_bounds, rotation_degrees, db_session)`
|
||||
|
||||
**Logic:**
|
||||
```
|
||||
1. Input: item_id, original_image_bytes, crop_bounds, rotation_degrees
|
||||
2. Check if crop_bounds and rotation_degrees are valid
|
||||
- If not: log warning, skip photo save, return success (graceful degradation)
|
||||
3. Create crop_bounds_dict from AI coordinates:
|
||||
{
|
||||
"x": crop_bounds["x"],
|
||||
"y": crop_bounds["y"],
|
||||
"w": crop_bounds["width"],
|
||||
"h": crop_bounds["height"]
|
||||
}
|
||||
4. Call ImageProcessor.process_photo(image_bytes, crop_bounds_dict, rotation_degrees)
|
||||
5. If processing fails: log error, skip photo save (don't block item creation)
|
||||
6. If processing succeeds:
|
||||
- Get unique filename using ImageStorage.get_unique_filename()
|
||||
- Save full image and thumbnail
|
||||
- Update Item.photo_path, photo_thumbnail_path, photo_upload_date
|
||||
7. Return: {status: "ok"} or {status: "skipped", reason: "..."}
|
||||
```
|
||||
|
||||
**Error Handling:**
|
||||
- Missing crop_bounds/rotation? → Skip photo save, item created successfully
|
||||
- Processing fails? → Log error, save original image as fallback
|
||||
- File save fails? → Log error, don't block item creation
|
||||
|
||||
---
|
||||
|
||||
## 4. Frontend Implementation
|
||||
|
||||
### 4.1 AIOnboarding Component Flow
|
||||
|
||||
**File:** `frontend/components/AIOnboarding.tsx`
|
||||
|
||||
**Current Flow:**
|
||||
1. Take photo
|
||||
2. Send to `/extract-label` (get item data only)
|
||||
3. Show extracted data, user edits
|
||||
4. User clicks "Create Item"
|
||||
5. Item created
|
||||
6. (Separate) User uploads photo later
|
||||
|
||||
**New Flow:**
|
||||
1. Take photo + store original image bytes in state
|
||||
2. Send to `/extract-label` (get item data + crop/rotation)
|
||||
3. Show extracted data + **store image_processing metadata** in state
|
||||
4. User edits item details
|
||||
5. User clicks "Create Item"
|
||||
6. **[NEW]** After item creation, auto-call `/items/{id}/photos` with:
|
||||
- file: original image
|
||||
- crop_bounds: from extraction response
|
||||
- replace_existing: "false"
|
||||
7. Show success toast: "Item created + photo saved"
|
||||
|
||||
### 4.2 Hook Updates
|
||||
|
||||
**File:** `frontend/hooks/useAIExtraction.ts`
|
||||
|
||||
**Changes:**
|
||||
- Store extracted `image_processing` data alongside item data
|
||||
- Pass to `useItemCreate` hook
|
||||
|
||||
**File:** `frontend/hooks/useItemCreate.ts`
|
||||
|
||||
**Changes:**
|
||||
- After item creation succeeds, check if `image_processing` exists
|
||||
- If yes: call `uploadPhoto()` with crop_bounds from extraction
|
||||
- Wait for photo upload to complete
|
||||
- Show combined toast: "Item + Photo saved"
|
||||
|
||||
### 4.3 Data Flow in State
|
||||
|
||||
```typescript
|
||||
// In AIOnboarding
|
||||
const [extractedImage, setExtractedImage] = useState<Blob | null>(null); // Original image
|
||||
const [imageProcessing, setImageProcessing] = useState<{crop_bounds, rotation_degrees, confidence} | null>(null);
|
||||
|
||||
// After extraction
|
||||
const response = await inventoryApi.analyzeLabel(formData, mode);
|
||||
setExtractedImage(imageBlob); // Store for later
|
||||
setImageProcessing(response.items[0].image_processing); // Store metadata
|
||||
|
||||
// When creating item, pass both to useItemCreate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Data Flow Diagram (Text)
|
||||
|
||||
```
|
||||
User takes photo
|
||||
↓
|
||||
POST /extract-label (with enhanced prompt)
|
||||
↓
|
||||
AI returns: {items: [{...item_data, image_processing: {crop_bounds, rotation_degrees, confidence}}]}
|
||||
↓
|
||||
Frontend stores: extracted_image + image_processing metadata
|
||||
↓
|
||||
Show item data, user edits
|
||||
↓
|
||||
User clicks "Create Item"
|
||||
↓
|
||||
POST /items → Item created in DB (item_id = 123)
|
||||
↓
|
||||
POST /items/123/photos with:
|
||||
- file: extracted_image
|
||||
- crop_bounds: JSON from image_processing
|
||||
- replace_existing: false
|
||||
↓
|
||||
Backend:
|
||||
1. Validate crop_bounds JSON
|
||||
2. Call ImageProcessor.process_photo(bytes, crop_bounds_dict)
|
||||
3. Save full image + thumbnail
|
||||
4. Update item.photo_path, photo_thumbnail_path, photo_upload_date
|
||||
↓
|
||||
Return success + photo URLs
|
||||
↓
|
||||
Show toast: "Item created + photo saved"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Files Modified
|
||||
|
||||
| File | Change | Lines |
|
||||
|------|--------|-------|
|
||||
| `config/ai_prompt.md` | Add "Image Processing Guidance" section with crop/rotation rules | +50 |
|
||||
| `backend/ai_vision.py` | Parse `image_processing` field from AI response | +20 |
|
||||
| `backend/routers/items.py` | Add `_auto_save_photo_from_extraction()` helper function; update item creation flow | +80 |
|
||||
| `frontend/hooks/useAIExtraction.ts` | Store `image_processing` metadata alongside extracted items | +15 |
|
||||
| `frontend/hooks/useItemCreate.ts` | Auto-call photo upload if `image_processing` exists after item creation | +30 |
|
||||
| `frontend/components/AIOnboarding.tsx` | Pass extracted image + image_processing to item creation | +10 |
|
||||
|
||||
**Total Impact:** ~205 lines of code/config
|
||||
|
||||
---
|
||||
|
||||
## 7. Error Handling & Fallbacks
|
||||
|
||||
| Scenario | Handling |
|
||||
|----------|----------|
|
||||
| AI doesn't return image_processing | Skip photo save, item created (no photo) |
|
||||
| crop_bounds is null/invalid | Skip photo save, item created (no photo) |
|
||||
| ImageProcessor.process_photo() fails | Log error, save original image as-is |
|
||||
| File save fails | Log error, don't block item creation |
|
||||
| Network error during photo upload | Return error to frontend (user can retry manually) |
|
||||
| User has no camera permission | Existing flow (file upload only) |
|
||||
|
||||
---
|
||||
|
||||
## 8. Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- Parse `image_processing` from AI response correctly
|
||||
- Validate crop_bounds JSON (x, y, width, height are valid)
|
||||
- Rotation degrees within valid range (-360 to +360)
|
||||
- Confidence score is 0.0-1.0
|
||||
|
||||
### Integration Tests
|
||||
- Extract → Create Item → Auto-save Photo flow end-to-end
|
||||
- Photo saved with correct crop/rotation applied
|
||||
- Fallback: photo save fails, item still created
|
||||
- Manual photo upload still works (separate flow)
|
||||
|
||||
### E2E Tests
|
||||
- User takes photo → AI extracts + crop guidance → creates item → photo auto-saved
|
||||
- Verify photo appears in inventory card with correct crop
|
||||
|
||||
---
|
||||
|
||||
## 9. Success Criteria
|
||||
|
||||
✅ Single API call returns item data + crop/rotation guidance
|
||||
✅ Backend applies crop/rotation from AI metadata
|
||||
✅ Photo auto-saved after item confirmation
|
||||
✅ No manual photo upload step needed for AI-identified items
|
||||
✅ Graceful fallback if processing fails
|
||||
✅ ~1000+ token savings per extraction (no image in response)
|
||||
✅ All existing tests pass
|
||||
✅ E2E test covers full flow
|
||||
|
||||
---
|
||||
|
||||
## 10. Rollout Strategy
|
||||
|
||||
**Phase 1:** Backend + Frontend changes (non-breaking)
|
||||
- Old `/extract-label` calls still work (image_processing field optional)
|
||||
- Manual photo upload still works
|
||||
- AIOnboarding auto-save only for new items with image_processing data
|
||||
|
||||
**Phase 2:** Update AI prompt in config (activate crop/rotation guidance)
|
||||
- Existing deployments get enhanced prompt on next config reload
|
||||
- New extractions return image_processing field
|
||||
|
||||
**Rollback:** Remove image_processing field from response, revert to manual upload
|
||||
|
||||
---
|
||||
|
||||
## 11. Notes
|
||||
|
||||
- **Backward Compatibility:** If AI doesn't return `image_processing`, system falls back to manual upload (no breaking change)
|
||||
- **Storage:** Original image passed from frontend to photo upload endpoint (already happens in current flow)
|
||||
- **Security:** No new endpoints, no new auth required (existing /extract-label and /items/{id}/photos endpoints)
|
||||
- **Performance:** Single AI call vs two API calls = 50% fewer round-trips
|
||||
Reference in New Issue
Block a user