Files

Daniel Bedeleanu ed42a9e306 docs: design AI extraction + auto-photo-save with crop/rotation guidance

2026-04-21 18:39:43 +03:00

11 KiB

Raw Blame History

Design: Single-Query AI Extraction + Auto-Photo-Save with Crop/Rotation

Date: 2026-04-21
Author: Claude Haiku 4.5
Status: Design Phase
Scope: Phase 3 - Photo Quality & Reliability

1. Overview

Problem:

Currently: Two separate API calls (extract-label for OCR, then upload-photo for crop/rotate)
Inefficient: Duplicate processing, higher token cost
User experience: Extra step after item creation to manually upload photo

Solution:

Single API call: /extract-label returns item data + crop/rotation metadata
Backend applies crop/rotation locally using AI guidance
Automatic photo save after item confirmation (no manual upload needed)
Token savings: ~1000+ tokens per item (no image in response, just coordinates)

Benefits:

50% fewer API calls
Single query to AI instead of two
Automatic photo integration (better UX)
Graceful fallback if processing fails

2. Enhanced AI Prompt

Current Prompt Structure

Extracts item data only (Item, Type, Description, Category, Connector, Size, Color, PartNr, OCR)
Returns JSON with extracted fields
No guidance on image layout or rotation

Enhanced Prompt Addition

Add to /config/ai_prompt.md (after Output Format section):

## Image Processing Guidance (NEW)

Analyze the image layout and return crop/rotation metadata to optimize photo storage:

### Crop Bounds Analysis
- Identify the PRIMARY ITEM in the image (main object, not background/clutter)
- Return bounding box: `{x, y, width, height}` in pixel coordinates
- Rules:
  - `x, y`: top-left corner of item (pixel offset from image top-left)
  - `width, height`: dimensions of item bounding box
  - Include minimal padding (10-15 pixels) around item edges
  - Ignore background clutter, other items, hands, reflections

### Rotation Analysis
- Check if item labels/text are readable
- If text is rotated (not horizontal), calculate rotation needed
- Return `rotation_degrees`: degrees to rotate CLOCKWISE to make text readable
- Examples:
  - Text rotated 90° counter-clockwise → return 90 (rotate 90° clockwise)
  - Text rotated 45° clockwise → return -45 (rotate 45° counter-clockwise)
  - Text already readable → return 0

### Confidence Score
- Return `confidence`: 0.0-1.0 indicating reliability of crop/rotation analysis
- 0.9+ = High confidence (clear item, readable text)
- 0.7-0.89 = Medium confidence (some ambiguity or text partially obscured)
- <0.7 = Low confidence (cluttered image, unclear item boundaries)

### Output Format (Extended)

```json
{
  "items": [
    {
      "Item": "[size] type vendor connector partnumber",
      "Type": "type",
      "Description": "technical details (max 5 words)",
      "Category": "category",
      "Connector": "connector_type",
      "Size": "human_readable_size",
      "Color": "color",
      "PartNr": "part_number",
      "OCR": "TYPE SIZE VENDOR CONNECTOR PARTNUMBER",
      "image_processing": {
        "crop_bounds": {
          "x": 50,
          "y": 100,
          "width": 300,
          "height": 200
        },
        "rotation_degrees": 15,
        "confidence": 0.92
      }
    }
  ]
}

Return ONLY JSON. No markdown. No text.


---

## 3. Backend Implementation

### 3.1 Updated Endpoint: `/extract-label`

**File:** `backend/routers/items.py`

**Changes:**
- Enhanced prompt now included in `ai_vision.extract_label_info()`
- AI response parsed to include `image_processing` field
- Returns crop_bounds, rotation_degrees, confidence

**Example Response:**
```json
{
  "items": [
    {
      "Item": "1.6TB NVMe HPE U.3 P66093-002",
      "Type": "NVMe",
      "Description": "Enterprise storage module",
      "Category": "Storage",
      "Connector": "U.3",
      "Size": "1.6TB",
      "Color": "Black",
      "PartNr": "P66093-002",
      "OCR": "NVME 1.6TB HPE U3 P66093002",
      "image_processing": {
        "crop_bounds": {"x": 45, "y": 80, "width": 350, "height": 220},
        "rotation_degrees": 12,
        "confidence": 0.94
      }
    }
  ]
}

3.2 Backend Photo Auto-Save Logic

File: backend/routers/items.py (new function)

Function: _auto_save_photo_from_extraction(item_id, image_bytes, crop_bounds, rotation_degrees, db_session)

Logic:

1. Input: item_id, original_image_bytes, crop_bounds, rotation_degrees
2. Check if crop_bounds and rotation_degrees are valid
   - If not: log warning, skip photo save, return success (graceful degradation)
3. Create crop_bounds_dict from AI coordinates:
   {
     "x": crop_bounds["x"],
     "y": crop_bounds["y"],
     "w": crop_bounds["width"],
     "h": crop_bounds["height"]
   }
4. Call ImageProcessor.process_photo(image_bytes, crop_bounds_dict, rotation_degrees)
5. If processing fails: log error, skip photo save (don't block item creation)
6. If processing succeeds:
   - Get unique filename using ImageStorage.get_unique_filename()
   - Save full image and thumbnail
   - Update Item.photo_path, photo_thumbnail_path, photo_upload_date
7. Return: {status: "ok"} or {status: "skipped", reason: "..."}

Error Handling:

Missing crop_bounds/rotation? → Skip photo save, item created successfully
Processing fails? → Log error, save original image as fallback
File save fails? → Log error, don't block item creation

4. Frontend Implementation

4.1 AIOnboarding Component Flow

File: frontend/components/AIOnboarding.tsx

Current Flow:

Take photo
Send to /extract-label (get item data only)
Show extracted data, user edits
User clicks "Create Item"
Item created
(Separate) User uploads photo later

New Flow:

Take photo + store original image bytes in state
Send to /extract-label (get item data + crop/rotation)
Show extracted data + store image_processing metadata in state
User edits item details
User clicks "Create Item"
[NEW] After item creation, auto-call /items/{id}/photos with:
- file: original image
- crop_bounds: from extraction response
- replace_existing: "false"
Show success toast: "Item created + photo saved"

4.2 Hook Updates

File: frontend/hooks/useAIExtraction.ts

Changes:

Store extracted image_processing data alongside item data
Pass to useItemCreate hook

File: frontend/hooks/useItemCreate.ts

Changes:

After item creation succeeds, check if image_processing exists
If yes: call uploadPhoto() with crop_bounds from extraction
Wait for photo upload to complete
Show combined toast: "Item + Photo saved"

4.3 Data Flow in State

// In AIOnboarding
const [extractedImage, setExtractedImage] = useState<Blob | null>(null);  // Original image
const [imageProcessing, setImageProcessing] = useState<{crop_bounds, rotation_degrees, confidence} | null>(null);

// After extraction
const response = await inventoryApi.analyzeLabel(formData, mode);
setExtractedImage(imageBlob);  // Store for later
setImageProcessing(response.items[0].image_processing);  // Store metadata

// When creating item, pass both to useItemCreate

5. Data Flow Diagram (Text)

User takes photo
    ↓
POST /extract-label (with enhanced prompt)
    ↓
AI returns: {items: [{...item_data, image_processing: {crop_bounds, rotation_degrees, confidence}}]}
    ↓
Frontend stores: extracted_image + image_processing metadata
    ↓
Show item data, user edits
    ↓
User clicks "Create Item"
    ↓
POST /items → Item created in DB (item_id = 123)
    ↓
POST /items/123/photos with:
  - file: extracted_image
  - crop_bounds: JSON from image_processing
  - replace_existing: false
    ↓
Backend:
  1. Validate crop_bounds JSON
  2. Call ImageProcessor.process_photo(bytes, crop_bounds_dict)
  3. Save full image + thumbnail
  4. Update item.photo_path, photo_thumbnail_path, photo_upload_date
    ↓
Return success + photo URLs
    ↓
Show toast: "Item created + photo saved"

6. Files Modified

File	Change	Lines
`config/ai_prompt.md`	Add "Image Processing Guidance" section with crop/rotation rules	+50
`backend/ai_vision.py`	Parse `image_processing` field from AI response	+20
`backend/routers/items.py`	Add `_auto_save_photo_from_extraction()` helper function; update item creation flow	+80
`frontend/hooks/useAIExtraction.ts`	Store `image_processing` metadata alongside extracted items	+15
`frontend/hooks/useItemCreate.ts`	Auto-call photo upload if `image_processing` exists after item creation	+30
`frontend/components/AIOnboarding.tsx`	Pass extracted image + image_processing to item creation	+10

Total Impact: ~205 lines of code/config

7. Error Handling & Fallbacks

Scenario	Handling
AI doesn't return image_processing	Skip photo save, item created (no photo)
crop_bounds is null/invalid	Skip photo save, item created (no photo)
ImageProcessor.process_photo() fails	Log error, save original image as-is
File save fails	Log error, don't block item creation
Network error during photo upload	Return error to frontend (user can retry manually)
User has no camera permission	Existing flow (file upload only)

8. Testing Strategy

Unit Tests

Parse image_processing from AI response correctly
Validate crop_bounds JSON (x, y, width, height are valid)
Rotation degrees within valid range (-360 to +360)
Confidence score is 0.0-1.0

Integration Tests

Extract → Create Item → Auto-save Photo flow end-to-end
Photo saved with correct crop/rotation applied
Fallback: photo save fails, item still created
Manual photo upload still works (separate flow)

E2E Tests

User takes photo → AI extracts + crop guidance → creates item → photo auto-saved
Verify photo appears in inventory card with correct crop

9. Success Criteria

✅ Single API call returns item data + crop/rotation guidance
✅ Backend applies crop/rotation from AI metadata
✅ Photo auto-saved after item confirmation
✅ No manual photo upload step needed for AI-identified items
✅ Graceful fallback if processing fails
✅ ~1000+ token savings per extraction (no image in response)
✅ All existing tests pass
✅ E2E test covers full flow

10. Rollout Strategy

Phase 1: Backend + Frontend changes (non-breaking)

Old /extract-label calls still work (image_processing field optional)
Manual photo upload still works
AIOnboarding auto-save only for new items with image_processing data

Phase 2: Update AI prompt in config (activate crop/rotation guidance)

Existing deployments get enhanced prompt on next config reload
New extractions return image_processing field

Rollback: Remove image_processing field from response, revert to manual upload

11. Notes

Backward Compatibility: If AI doesn't return image_processing, system falls back to manual upload (no breaking change)
Storage: Original image passed from frontend to photo upload endpoint (already happens in current flow)
Security: No new endpoints, no new auth required (existing /extract-label and /items/{id}/photos endpoints)
Performance: Single AI call vs two API calls = 50% fewer round-trips

11 KiB Raw Blame History