351 lines
11 KiB
Markdown
351 lines
11 KiB
Markdown
# Design: Single-Query AI Extraction + Auto-Photo-Save with Crop/Rotation
|
|
|
|
**Date:** 2026-04-21
|
|
**Author:** Claude Haiku 4.5
|
|
**Status:** Design Phase
|
|
**Scope:** Phase 3 - Photo Quality & Reliability
|
|
|
|
---
|
|
|
|
## 1. Overview
|
|
|
|
**Problem:**
|
|
- Currently: Two separate API calls (extract-label for OCR, then upload-photo for crop/rotate)
|
|
- Inefficient: Duplicate processing, higher token cost
|
|
- User experience: Extra step after item creation to manually upload photo
|
|
|
|
**Solution:**
|
|
- Single API call: `/extract-label` returns item data + crop/rotation metadata
|
|
- Backend applies crop/rotation locally using AI guidance
|
|
- Automatic photo save after item confirmation (no manual upload needed)
|
|
- **Token savings:** ~1000+ tokens per item (no image in response, just coordinates)
|
|
|
|
**Benefits:**
|
|
- 50% fewer API calls
|
|
- Single query to AI instead of two
|
|
- Automatic photo integration (better UX)
|
|
- Graceful fallback if processing fails
|
|
|
|
---
|
|
|
|
## 2. Enhanced AI Prompt
|
|
|
|
### Current Prompt Structure
|
|
- Extracts item data only (Item, Type, Description, Category, Connector, Size, Color, PartNr, OCR)
|
|
- Returns JSON with extracted fields
|
|
- No guidance on image layout or rotation
|
|
|
|
### Enhanced Prompt Addition
|
|
|
|
Add to `/config/ai_prompt.md` (after Output Format section):
|
|
|
|
```markdown
|
|
## Image Processing Guidance (NEW)
|
|
|
|
Analyze the image layout and return crop/rotation metadata to optimize photo storage:
|
|
|
|
### Crop Bounds Analysis
|
|
- Identify the PRIMARY ITEM in the image (main object, not background/clutter)
|
|
- Return bounding box: `{x, y, width, height}` in pixel coordinates
|
|
- Rules:
|
|
- `x, y`: top-left corner of item (pixel offset from image top-left)
|
|
- `width, height`: dimensions of item bounding box
|
|
- Include minimal padding (10-15 pixels) around item edges
|
|
- Ignore background clutter, other items, hands, reflections
|
|
|
|
### Rotation Analysis
|
|
- Check if item labels/text are readable
|
|
- If text is rotated (not horizontal), calculate rotation needed
|
|
- Return `rotation_degrees`: degrees to rotate CLOCKWISE to make text readable
|
|
- Examples:
|
|
- Text rotated 90° counter-clockwise → return 90 (rotate 90° clockwise)
|
|
- Text rotated 45° clockwise → return -45 (rotate 45° counter-clockwise)
|
|
- Text already readable → return 0
|
|
|
|
### Confidence Score
|
|
- Return `confidence`: 0.0-1.0 indicating reliability of crop/rotation analysis
|
|
- 0.9+ = High confidence (clear item, readable text)
|
|
- 0.7-0.89 = Medium confidence (some ambiguity or text partially obscured)
|
|
- <0.7 = Low confidence (cluttered image, unclear item boundaries)
|
|
|
|
### Output Format (Extended)
|
|
|
|
```json
|
|
{
|
|
"items": [
|
|
{
|
|
"Item": "[size] type vendor connector partnumber",
|
|
"Type": "type",
|
|
"Description": "technical details (max 5 words)",
|
|
"Category": "category",
|
|
"Connector": "connector_type",
|
|
"Size": "human_readable_size",
|
|
"Color": "color",
|
|
"PartNr": "part_number",
|
|
"OCR": "TYPE SIZE VENDOR CONNECTOR PARTNUMBER",
|
|
"image_processing": {
|
|
"crop_bounds": {
|
|
"x": 50,
|
|
"y": 100,
|
|
"width": 300,
|
|
"height": 200
|
|
},
|
|
"rotation_degrees": 15,
|
|
"confidence": 0.92
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
**Return ONLY JSON. No markdown. No text.**
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Backend Implementation
|
|
|
|
### 3.1 Updated Endpoint: `/extract-label`
|
|
|
|
**File:** `backend/routers/items.py`
|
|
|
|
**Changes:**
|
|
- Enhanced prompt now included in `ai_vision.extract_label_info()`
|
|
- AI response parsed to include `image_processing` field
|
|
- Returns crop_bounds, rotation_degrees, confidence
|
|
|
|
**Example Response:**
|
|
```json
|
|
{
|
|
"items": [
|
|
{
|
|
"Item": "1.6TB NVMe HPE U.3 P66093-002",
|
|
"Type": "NVMe",
|
|
"Description": "Enterprise storage module",
|
|
"Category": "Storage",
|
|
"Connector": "U.3",
|
|
"Size": "1.6TB",
|
|
"Color": "Black",
|
|
"PartNr": "P66093-002",
|
|
"OCR": "NVME 1.6TB HPE U3 P66093002",
|
|
"image_processing": {
|
|
"crop_bounds": {"x": 45, "y": 80, "width": 350, "height": 220},
|
|
"rotation_degrees": 12,
|
|
"confidence": 0.94
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### 3.2 Backend Photo Auto-Save Logic
|
|
|
|
**File:** `backend/routers/items.py` (new function)
|
|
|
|
**Function:** `_auto_save_photo_from_extraction(item_id, image_bytes, crop_bounds, rotation_degrees, db_session)`
|
|
|
|
**Logic:**
|
|
```
|
|
1. Input: item_id, original_image_bytes, crop_bounds, rotation_degrees
|
|
2. Check if crop_bounds and rotation_degrees are valid
|
|
- If not: log warning, skip photo save, return success (graceful degradation)
|
|
3. Create crop_bounds_dict from AI coordinates:
|
|
{
|
|
"x": crop_bounds["x"],
|
|
"y": crop_bounds["y"],
|
|
"w": crop_bounds["width"],
|
|
"h": crop_bounds["height"]
|
|
}
|
|
4. Call ImageProcessor.process_photo(image_bytes, crop_bounds_dict, rotation_degrees)
|
|
5. If processing fails: log error, skip photo save (don't block item creation)
|
|
6. If processing succeeds:
|
|
- Get unique filename using ImageStorage.get_unique_filename()
|
|
- Save full image and thumbnail
|
|
- Update Item.photo_path, photo_thumbnail_path, photo_upload_date
|
|
7. Return: {status: "ok"} or {status: "skipped", reason: "..."}
|
|
```
|
|
|
|
**Error Handling:**
|
|
- Missing crop_bounds/rotation? → Skip photo save, item created successfully
|
|
- Processing fails? → Log error, save original image as fallback
|
|
- File save fails? → Log error, don't block item creation
|
|
|
|
---
|
|
|
|
## 4. Frontend Implementation
|
|
|
|
### 4.1 AIOnboarding Component Flow
|
|
|
|
**File:** `frontend/components/AIOnboarding.tsx`
|
|
|
|
**Current Flow:**
|
|
1. Take photo
|
|
2. Send to `/extract-label` (get item data only)
|
|
3. Show extracted data, user edits
|
|
4. User clicks "Create Item"
|
|
5. Item created
|
|
6. (Separate) User uploads photo later
|
|
|
|
**New Flow:**
|
|
1. Take photo + store original image bytes in state
|
|
2. Send to `/extract-label` (get item data + crop/rotation)
|
|
3. Show extracted data + **store image_processing metadata** in state
|
|
4. User edits item details
|
|
5. User clicks "Create Item"
|
|
6. **[NEW]** After item creation, auto-call `/items/{id}/photos` with:
|
|
- file: original image
|
|
- crop_bounds: from extraction response
|
|
- replace_existing: "false"
|
|
7. Show success toast: "Item created + photo saved"
|
|
|
|
### 4.2 Hook Updates
|
|
|
|
**File:** `frontend/hooks/useAIExtraction.ts`
|
|
|
|
**Changes:**
|
|
- Store extracted `image_processing` data alongside item data
|
|
- Pass to `useItemCreate` hook
|
|
|
|
**File:** `frontend/hooks/useItemCreate.ts`
|
|
|
|
**Changes:**
|
|
- After item creation succeeds, check if `image_processing` exists
|
|
- If yes: call `uploadPhoto()` with crop_bounds from extraction
|
|
- Wait for photo upload to complete
|
|
- Show combined toast: "Item + Photo saved"
|
|
|
|
### 4.3 Data Flow in State
|
|
|
|
```typescript
|
|
// In AIOnboarding
|
|
const [extractedImage, setExtractedImage] = useState<Blob | null>(null); // Original image
|
|
const [imageProcessing, setImageProcessing] = useState<{crop_bounds, rotation_degrees, confidence} | null>(null);
|
|
|
|
// After extraction
|
|
const response = await inventoryApi.analyzeLabel(formData, mode);
|
|
setExtractedImage(imageBlob); // Store for later
|
|
setImageProcessing(response.items[0].image_processing); // Store metadata
|
|
|
|
// When creating item, pass both to useItemCreate
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Data Flow Diagram (Text)
|
|
|
|
```
|
|
User takes photo
|
|
↓
|
|
POST /extract-label (with enhanced prompt)
|
|
↓
|
|
AI returns: {items: [{...item_data, image_processing: {crop_bounds, rotation_degrees, confidence}}]}
|
|
↓
|
|
Frontend stores: extracted_image + image_processing metadata
|
|
↓
|
|
Show item data, user edits
|
|
↓
|
|
User clicks "Create Item"
|
|
↓
|
|
POST /items → Item created in DB (item_id = 123)
|
|
↓
|
|
POST /items/123/photos with:
|
|
- file: extracted_image
|
|
- crop_bounds: JSON from image_processing
|
|
- replace_existing: false
|
|
↓
|
|
Backend:
|
|
1. Validate crop_bounds JSON
|
|
2. Call ImageProcessor.process_photo(bytes, crop_bounds_dict)
|
|
3. Save full image + thumbnail
|
|
4. Update item.photo_path, photo_thumbnail_path, photo_upload_date
|
|
↓
|
|
Return success + photo URLs
|
|
↓
|
|
Show toast: "Item created + photo saved"
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Files Modified
|
|
|
|
| File | Change | Lines |
|
|
|------|--------|-------|
|
|
| `config/ai_prompt.md` | Add "Image Processing Guidance" section with crop/rotation rules | +50 |
|
|
| `backend/ai_vision.py` | Parse `image_processing` field from AI response | +20 |
|
|
| `backend/routers/items.py` | Add `_auto_save_photo_from_extraction()` helper function; update item creation flow | +80 |
|
|
| `frontend/hooks/useAIExtraction.ts` | Store `image_processing` metadata alongside extracted items | +15 |
|
|
| `frontend/hooks/useItemCreate.ts` | Auto-call photo upload if `image_processing` exists after item creation | +30 |
|
|
| `frontend/components/AIOnboarding.tsx` | Pass extracted image + image_processing to item creation | +10 |
|
|
|
|
**Total Impact:** ~205 lines of code/config
|
|
|
|
---
|
|
|
|
## 7. Error Handling & Fallbacks
|
|
|
|
| Scenario | Handling |
|
|
|----------|----------|
|
|
| AI doesn't return image_processing | Skip photo save, item created (no photo) |
|
|
| crop_bounds is null/invalid | Skip photo save, item created (no photo) |
|
|
| ImageProcessor.process_photo() fails | Log error, save original image as-is |
|
|
| File save fails | Log error, don't block item creation |
|
|
| Network error during photo upload | Return error to frontend (user can retry manually) |
|
|
| User has no camera permission | Existing flow (file upload only) |
|
|
|
|
---
|
|
|
|
## 8. Testing Strategy
|
|
|
|
### Unit Tests
|
|
- Parse `image_processing` from AI response correctly
|
|
- Validate crop_bounds JSON (x, y, width, height are valid)
|
|
- Rotation degrees within valid range (-360 to +360)
|
|
- Confidence score is 0.0-1.0
|
|
|
|
### Integration Tests
|
|
- Extract → Create Item → Auto-save Photo flow end-to-end
|
|
- Photo saved with correct crop/rotation applied
|
|
- Fallback: photo save fails, item still created
|
|
- Manual photo upload still works (separate flow)
|
|
|
|
### E2E Tests
|
|
- User takes photo → AI extracts + crop guidance → creates item → photo auto-saved
|
|
- Verify photo appears in inventory card with correct crop
|
|
|
|
---
|
|
|
|
## 9. Success Criteria
|
|
|
|
✅ Single API call returns item data + crop/rotation guidance
|
|
✅ Backend applies crop/rotation from AI metadata
|
|
✅ Photo auto-saved after item confirmation
|
|
✅ No manual photo upload step needed for AI-identified items
|
|
✅ Graceful fallback if processing fails
|
|
✅ ~1000+ token savings per extraction (no image in response)
|
|
✅ All existing tests pass
|
|
✅ E2E test covers full flow
|
|
|
|
---
|
|
|
|
## 10. Rollout Strategy
|
|
|
|
**Phase 1:** Backend + Frontend changes (non-breaking)
|
|
- Old `/extract-label` calls still work (image_processing field optional)
|
|
- Manual photo upload still works
|
|
- AIOnboarding auto-save only for new items with image_processing data
|
|
|
|
**Phase 2:** Update AI prompt in config (activate crop/rotation guidance)
|
|
- Existing deployments get enhanced prompt on next config reload
|
|
- New extractions return image_processing field
|
|
|
|
**Rollback:** Remove image_processing field from response, revert to manual upload
|
|
|
|
---
|
|
|
|
## 11. Notes
|
|
|
|
- **Backward Compatibility:** If AI doesn't return `image_processing`, system falls back to manual upload (no breaking change)
|
|
- **Storage:** Original image passed from frontend to photo upload endpoint (already happens in current flow)
|
|
- **Security:** No new endpoints, no new auth required (existing /extract-label and /items/{id}/photos endpoints)
|
|
- **Performance:** Single AI call vs two API calls = 50% fewer round-trips
|