# Design: Single-Query AI Extraction + Auto-Photo-Save with Crop/Rotation

**Date:** 2026-04-21  
**Author:** Claude Haiku 4.5  
**Status:** Design Phase  
**Scope:** Phase 3 - Photo Quality & Reliability

---

## 1. Overview

**Problem:**
- Currently: Two separate API calls (extract-label for OCR, then upload-photo for crop/rotate)
- Inefficient: Duplicate processing, higher token cost
- User experience: Extra step after item creation to manually upload photo

**Solution:**
- Single API call: `/extract-label` returns item data + crop/rotation metadata
- Backend applies crop/rotation locally using AI guidance
- Automatic photo save after item confirmation (no manual upload needed)
- **Token savings:** ~1000+ tokens per item (no image in response, just coordinates)

**Benefits:**
- 50% fewer API calls
- Single query to AI instead of two
- Automatic photo integration (better UX)
- Graceful fallback if processing fails

---

## 2. Enhanced AI Prompt

### Current Prompt Structure
- Extracts item data only (Item, Type, Description, Category, Connector, Size, Color, PartNr, OCR)
- Returns JSON with extracted fields
- No guidance on image layout or rotation

### Enhanced Prompt Addition

Add to `/config/ai_prompt.md` (after Output Format section):

```markdown
## Image Processing Guidance (NEW)

Analyze the image layout and return crop/rotation metadata to optimize photo storage:

### Crop Bounds Analysis
- Identify the PRIMARY ITEM in the image (main object, not background/clutter)
- Return bounding box: `{x, y, width, height}` in pixel coordinates
- Rules:
  - `x, y`: top-left corner of item (pixel offset from image top-left)
  - `width, height`: dimensions of item bounding box
  - Include minimal padding (10-15 pixels) around item edges
  - Ignore background clutter, other items, hands, reflections

### Rotation Analysis
- Check if item labels/text are readable
- If text is rotated (not horizontal), calculate rotation needed
- Return `rotation_degrees`: degrees to rotate CLOCKWISE to make text readable
- Examples:
  - Text rotated 90° counter-clockwise → return 90 (rotate 90° clockwise)
  - Text rotated 45° clockwise → return -45 (rotate 45° counter-clockwise)
  - Text already readable → return 0

### Confidence Score
- Return `confidence`: 0.0-1.0 indicating reliability of crop/rotation analysis
- 0.9+ = High confidence (clear item, readable text)
- 0.7-0.89 = Medium confidence (some ambiguity or text partially obscured)
- <0.7 = Low confidence (cluttered image, unclear item boundaries)

### Output Format (Extended)

```json
{
  "items": [
    {
      "Item": "[size] type vendor connector partnumber",
      "Type": "type",
      "Description": "technical details (max 5 words)",
      "Category": "category",
      "Connector": "connector_type",
      "Size": "human_readable_size",
      "Color": "color",
      "PartNr": "part_number",
      "OCR": "TYPE SIZE VENDOR CONNECTOR PARTNUMBER",
      "image_processing": {
        "crop_bounds": {
          "x": 50,
          "y": 100,
          "width": 300,
          "height": 200
        },
        "rotation_degrees": 15,
        "confidence": 0.92
      }
    }
  ]
}
```

**Return ONLY JSON. No markdown. No text.**
```

---

## 3. Backend Implementation

### 3.1 Updated Endpoint: `/extract-label`

**File:** `backend/routers/items.py`

**Changes:**
- Enhanced prompt now included in `ai_vision.extract_label_info()`
- AI response parsed to include `image_processing` field
- Returns crop_bounds, rotation_degrees, confidence

**Example Response:**
```json
{
  "items": [
    {
      "Item": "1.6TB NVMe HPE U.3 P66093-002",
      "Type": "NVMe",
      "Description": "Enterprise storage module",
      "Category": "Storage",
      "Connector": "U.3",
      "Size": "1.6TB",
      "Color": "Black",
      "PartNr": "P66093-002",
      "OCR": "NVME 1.6TB HPE U3 P66093002",
      "image_processing": {
        "crop_bounds": {"x": 45, "y": 80, "width": 350, "height": 220},
        "rotation_degrees": 12,
        "confidence": 0.94
      }
    }
  ]
}
```

### 3.2 Backend Photo Auto-Save Logic

**File:** `backend/routers/items.py` (new function)

**Function:** `_auto_save_photo_from_extraction(item_id, image_bytes, crop_bounds, rotation_degrees, db_session)`

**Logic:**
```
1. Input: item_id, original_image_bytes, crop_bounds, rotation_degrees
2. Check if crop_bounds and rotation_degrees are valid
   - If not: log warning, skip photo save, return success (graceful degradation)
3. Create crop_bounds_dict from AI coordinates:
   {
     "x": crop_bounds["x"],
     "y": crop_bounds["y"],
     "w": crop_bounds["width"],
     "h": crop_bounds["height"]
   }
4. Call ImageProcessor.process_photo(image_bytes, crop_bounds_dict, rotation_degrees)
5. If processing fails: log error, skip photo save (don't block item creation)
6. If processing succeeds:
   - Get unique filename using ImageStorage.get_unique_filename()
   - Save full image and thumbnail
   - Update Item.photo_path, photo_thumbnail_path, photo_upload_date
7. Return: {status: "ok"} or {status: "skipped", reason: "..."}
```

**Error Handling:**
- Missing crop_bounds/rotation? → Skip photo save, item created successfully
- Processing fails? → Log error, save original image as fallback
- File save fails? → Log error, don't block item creation

---

## 4. Frontend Implementation

### 4.1 AIOnboarding Component Flow

**File:** `frontend/components/AIOnboarding.tsx`

**Current Flow:**
1. Take photo
2. Send to `/extract-label` (get item data only)
3. Show extracted data, user edits
4. User clicks "Create Item"
5. Item created
6. (Separate) User uploads photo later

**New Flow:**
1. Take photo + store original image bytes in state
2. Send to `/extract-label` (get item data + crop/rotation)
3. Show extracted data + **store image_processing metadata** in state
4. User edits item details
5. User clicks "Create Item"
6. **[NEW]** After item creation, auto-call `/items/{id}/photos` with:
   - file: original image
   - crop_bounds: from extraction response
   - replace_existing: "false"
7. Show success toast: "Item created + photo saved"

### 4.2 Hook Updates

**File:** `frontend/hooks/useAIExtraction.ts`

**Changes:**
- Store extracted `image_processing` data alongside item data
- Pass to `useItemCreate` hook

**File:** `frontend/hooks/useItemCreate.ts`

**Changes:**
- After item creation succeeds, check if `image_processing` exists
- If yes: call `uploadPhoto()` with crop_bounds from extraction
- Wait for photo upload to complete
- Show combined toast: "Item + Photo saved"

### 4.3 Data Flow in State

```typescript
// In AIOnboarding
const [extractedImage, setExtractedImage] = useState<Blob | null>(null);  // Original image
const [imageProcessing, setImageProcessing] = useState<{crop_bounds, rotation_degrees, confidence} | null>(null);

// After extraction
const response = await inventoryApi.analyzeLabel(formData, mode);
setExtractedImage(imageBlob);  // Store for later
setImageProcessing(response.items[0].image_processing);  // Store metadata

// When creating item, pass both to useItemCreate
```

---

## 5. Data Flow Diagram (Text)

```
User takes photo
    ↓
POST /extract-label (with enhanced prompt)
    ↓
AI returns: {items: [{...item_data, image_processing: {crop_bounds, rotation_degrees, confidence}}]}
    ↓
Frontend stores: extracted_image + image_processing metadata
    ↓
Show item data, user edits
    ↓
User clicks "Create Item"
    ↓
POST /items → Item created in DB (item_id = 123)
    ↓
POST /items/123/photos with:
  - file: extracted_image
  - crop_bounds: JSON from image_processing
  - replace_existing: false
    ↓
Backend:
  1. Validate crop_bounds JSON
  2. Call ImageProcessor.process_photo(bytes, crop_bounds_dict)
  3. Save full image + thumbnail
  4. Update item.photo_path, photo_thumbnail_path, photo_upload_date
    ↓
Return success + photo URLs
    ↓
Show toast: "Item created + photo saved"
```

---

## 6. Files Modified

| File | Change | Lines |
|------|--------|-------|
| `config/ai_prompt.md` | Add "Image Processing Guidance" section with crop/rotation rules | +50 |
| `backend/ai_vision.py` | Parse `image_processing` field from AI response | +20 |
| `backend/routers/items.py` | Add `_auto_save_photo_from_extraction()` helper function; update item creation flow | +80 |
| `frontend/hooks/useAIExtraction.ts` | Store `image_processing` metadata alongside extracted items | +15 |
| `frontend/hooks/useItemCreate.ts` | Auto-call photo upload if `image_processing` exists after item creation | +30 |
| `frontend/components/AIOnboarding.tsx` | Pass extracted image + image_processing to item creation | +10 |

**Total Impact:** ~205 lines of code/config

---

## 7. Error Handling & Fallbacks

| Scenario | Handling |
|----------|----------|
| AI doesn't return image_processing | Skip photo save, item created (no photo) |
| crop_bounds is null/invalid | Skip photo save, item created (no photo) |
| ImageProcessor.process_photo() fails | Log error, save original image as-is |
| File save fails | Log error, don't block item creation |
| Network error during photo upload | Return error to frontend (user can retry manually) |
| User has no camera permission | Existing flow (file upload only) |

---

## 8. Testing Strategy

### Unit Tests
- Parse `image_processing` from AI response correctly
- Validate crop_bounds JSON (x, y, width, height are valid)
- Rotation degrees within valid range (-360 to +360)
- Confidence score is 0.0-1.0

### Integration Tests
- Extract → Create Item → Auto-save Photo flow end-to-end
- Photo saved with correct crop/rotation applied
- Fallback: photo save fails, item still created
- Manual photo upload still works (separate flow)

### E2E Tests
- User takes photo → AI extracts + crop guidance → creates item → photo auto-saved
- Verify photo appears in inventory card with correct crop

---

## 9. Success Criteria

✅ Single API call returns item data + crop/rotation guidance  
✅ Backend applies crop/rotation from AI metadata  
✅ Photo auto-saved after item confirmation  
✅ No manual photo upload step needed for AI-identified items  
✅ Graceful fallback if processing fails  
✅ ~1000+ token savings per extraction (no image in response)  
✅ All existing tests pass  
✅ E2E test covers full flow  

---

## 10. Rollout Strategy

**Phase 1:** Backend + Frontend changes (non-breaking)
- Old `/extract-label` calls still work (image_processing field optional)
- Manual photo upload still works
- AIOnboarding auto-save only for new items with image_processing data

**Phase 2:** Update AI prompt in config (activate crop/rotation guidance)
- Existing deployments get enhanced prompt on next config reload
- New extractions return image_processing field

**Rollback:** Remove image_processing field from response, revert to manual upload

---

## 11. Notes

- **Backward Compatibility:** If AI doesn't return `image_processing`, system falls back to manual upload (no breaking change)
- **Storage:** Original image passed from frontend to photo upload endpoint (already happens in current flow)
- **Security:** No new endpoints, no new auth required (existing /extract-label and /items/{id}/photos endpoints)
- **Performance:** Single AI call vs two API calls = 50% fewer round-trips