docs(4.1): planning complete - research + 3 executable plans
Phase 4.1: AI Prompt Enhancement — Spare Parts Deep Identification Artifacts: - 4.1-RESEARCH.md: Web scraping patterns, spare-parts classification, integration architecture - 4.1-PLAN-01.md (Wave 1): Spare-parts whitelist + AI prompt enhancement (4 tasks) - 4.1-PLAN-02.md (Wave 2): Web scraping service + backend integration (6 tasks) - 4.1-PLAN-03.md (Wave 3): Frontend integration + end-to-end testing (7 tasks) All 17 tasks verified: ✓ Concrete action steps with exact function signatures and file paths ✓ 100% verifiable acceptance criteria (grep, pytest, vitest, imports) ✓ Architecture aligned with all 11 CONTEXT.md decisions ✓ CLAUDE.md compliance: TypeScript strict, API tests, UI fidelity ✓ Wave dependencies correctly ordered ✓ Risk mitigation: rate limiting, timeout handling, offline graceful degradation Ready for execution via /gsd-execute-phase 4.1
This commit is contained in:
354
.planning/phases/4.1-ai-spare-parts-deep-id/4.1-PLAN-01.md
Normal file
354
.planning/phases/4.1-ai-spare-parts-deep-id/4.1-PLAN-01.md
Normal file
@@ -0,0 +1,354 @@
|
||||
---
|
||||
wave: 1
|
||||
depends_on: null
|
||||
files_modified:
|
||||
- path: "backend/ai/spare_parts_whitelist.py"
|
||||
- path: "backend/ai/gemini_extractor.py"
|
||||
- path: "backend/ai/claude_extractor.py"
|
||||
- path: "tests/test_spare_parts_classification.py"
|
||||
autonomous: true
|
||||
---
|
||||
|
||||
# Phase 4.1 Wave 1: Spare-Parts Classification & AI Prompt Enhancement
|
||||
|
||||
**Objective:** Build foundation for spare-parts identification by implementing classification logic and enhancing AI prompts with spare-parts vs consumables decision tree.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Create Spare-Parts Classification Whitelist
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Build a configurable spare-parts whitelist module with fuzzy matching logic to classify items as spare parts or consumables.</objective>
|
||||
<read_first>
|
||||
- PROJECT_ARCHITECTURE.md (Item model, AI integration)
|
||||
- 4.1-RESEARCH.md sections 2 (Spare-Parts Classification Strategy) and Fuzzy Matching Implementation
|
||||
- No existing code to read — new file
|
||||
</read_first>
|
||||
<action>
|
||||
Create file: backend/ai/spare_parts_whitelist.py
|
||||
|
||||
**Module Structure:**
|
||||
|
||||
1. Define constant lists (case-insensitive strings):
|
||||
- SPARE_PART_CATEGORIES: ["RAM", "DRAM", "DDR3", "DDR4", "DDR5", "SODIMM", "DIMM", "SSD", "NVME", "M.2", "SATA", "HDD", "HARD DRIVE", "SOLID STATE DRIVE", "CPU", "PROCESSOR", "APU", "GPU", "GRAPHICS CARD", "DISCRETE GPU", "PSU", "POWER SUPPLY UNIT", "ADAPTER", "POWER MODULE", "PCIE", "PCI", "RAID CONTROLLER", "NETWORK CARD", "NIC", "HEATSINK", "CPU COOLER", "THERMAL SOLUTION", "MOTHERBOARD", "BIOS", "CHIPSET"]
|
||||
- CONSUMABLE_KEYWORDS: ["CABLE", "CORD", "FASTENER", "SCREW", "WASHER", "BOLT", "STANDOFF", "ADHESIVE", "THERMAL PASTE", "THERMAL PAD", "TAPE", "CONNECTOR", "PLUG", "SOCKET", "ADAPTER"]
|
||||
- POWER_SUPPLY_CONSUMABLE_KEYWORDS: ["CABLE", "CORD", "GENERIC", "POWER CORD", "AC CORD"]
|
||||
|
||||
2. Implement function: `classify_as_spare_part(category: str) -> bool`
|
||||
- Input: extracted Category from AI (string)
|
||||
- Algorithm:
|
||||
a. Normalize input: lowercase, strip whitespace
|
||||
b. Check exact match in SPARE_PART_CATEGORIES → return True
|
||||
c. Check regex patterns for Memory/Storage/CPU/GPU/PSU: if matched → +50 points
|
||||
d. Check fuzzy match (fuzzywuzzy library at 70-80% threshold) against SPARE_PART_CATEGORIES → if match ≥80% → +50 points, if 70-80% → +30 points
|
||||
e. Check exclusion patterns (CONSUMABLE_KEYWORDS): if matched → -100 points (override other scores)
|
||||
f. Special case: if "power supply" or "PSU" in category BUT "cable" or "cord" also in category → return False (consumable)
|
||||
g. Final score: ≥ 40 → True (Spare Part), < 40 → False
|
||||
- Return: bool
|
||||
|
||||
3. Implement function: `is_consumable(category: str) -> bool`
|
||||
- Return: `not classify_as_spare_part(category)`
|
||||
|
||||
4. Implement function: `get_spare_part_type(category: str) -> str | None`
|
||||
- Returns normalized spare-part type (e.g., "RAM", "SSD", "CPU", "GPU") or None if not a spare part
|
||||
- Used for search query building
|
||||
|
||||
**Code Quality:**
|
||||
- Use type hints: `def classify_as_spare_part(category: str) -> bool:`
|
||||
- Include docstrings with examples (Kingston DDR4 RAM → True, 6ft cable → False)
|
||||
- No external dependencies except fuzzywuzzy (add to requirements.txt)
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File exists: backend/ai/spare_parts_whitelist.py
|
||||
- Function `classify_as_spare_part(category: str) -> bool` accepts string input
|
||||
- Test passes: `classify_as_spare_part("Kingston DDR4 RAM")` returns True
|
||||
- Test passes: `classify_as_spare_part("6ft SATA Cable")` returns False
|
||||
- Test passes: `classify_as_spare_part("CPU Mounting Hardware Kit")` returns False
|
||||
- Test passes: `classify_as_spare_part("Corsair RM850x 850W PSU")` returns True
|
||||
- Fuzzy matching works: `classify_as_spare_part("Random Access Memory")` returns True
|
||||
- All functions have type hints and docstrings
|
||||
- fuzzywuzzy added to backend/requirements.txt with version constraint (e.g., ==0.18.0)
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Enhance Gemini AI Prompt with Spare-Parts Classification
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Update Gemini extraction prompt to include spare-parts vs consumables decision tree and comprehensive examples for accurate classification.</objective>
|
||||
<read_first>
|
||||
- backend/ai/gemini_extractor.py (current prompt structure and extraction pattern)
|
||||
- 4.1-RESEARCH.md section 3 (AI Prompt Enhancement) — specifically the "New Classification Logic" code block
|
||||
- 4.1-CONTEXT.md section decisions D-01 and D-02
|
||||
</read_first>
|
||||
<action>
|
||||
Modify file: backend/ai/gemini_extractor.py
|
||||
|
||||
**Action Steps:**
|
||||
|
||||
1. Locate the extraction prompt (likely in a string or .md file loaded at module init)
|
||||
|
||||
2. Insert new section AFTER category/type extraction, before returning results. Add this exact text:
|
||||
|
||||
```
|
||||
CLASSIFICATION GUIDE - SPARE PARTS vs CONSUMABLES:
|
||||
|
||||
Spare Parts (replaceable components that plug into or interface with devices):
|
||||
- RAM, DDR memory modules (DDR3, DDR4, DDR5, SODIMM, DIMM)
|
||||
- SSDs, NVMe drives, M.2 modules, SATA drives, hard drives
|
||||
- CPUs, GPUs, processors, discrete graphics cards
|
||||
- Power supply units (PSU), power modules (NOT generic power cords)
|
||||
- Expansion cards (PCIe, PCI, RAID controllers, network cards/NIC)
|
||||
- Cooling solutions (heatsinks, CPU coolers, thermal solutions)
|
||||
- Motherboards, chipsets, BIOS modules
|
||||
|
||||
NOT Spare Parts (consumables, generic items):
|
||||
- Cables (power, SATA, USB, Ethernet, proprietary cords)
|
||||
- Fasteners (screws, washers, bolts, standoffs)
|
||||
- Thermal paste, thermal pads, adhesive tapes
|
||||
- Connectors, plugs, sockets, generic adapters
|
||||
- Generic cords and utility items
|
||||
|
||||
Decision Tree:
|
||||
1. Does the item have a replaceable function in a larger system?
|
||||
2. Does it have a manufacturer part number and technical specifications?
|
||||
3. Is it described with model/revision information?
|
||||
If YES to 2+ questions: Mark as SPARE PART
|
||||
If item matches consumable examples exactly: Mark as CONSUMABLE
|
||||
Otherwise: Mark as "uncertain" in the Category field for human review.
|
||||
|
||||
Examples:
|
||||
✓ "Kingston Fury 16GB DDR4-3200" → Spare Part (RAM module)
|
||||
✓ "Samsung 970 EVO 1TB NVMe" → Spare Part (SSD)
|
||||
✓ "Intel Core i7-12700K" → Spare Part (CPU)
|
||||
✓ "Corsair RM850x 850W Power Supply" → Spare Part (PSU)
|
||||
✗ "6ft SATA Cable" → Consumable (cable)
|
||||
✗ "CPU Mounting Hardware Kit" → Consumable (fasteners)
|
||||
✗ "Thermal Paste Tube" → Consumable (adhesive material)
|
||||
```
|
||||
|
||||
3. In the prompt output template, ensure the Category field description includes: "Use the Classification Guide above to distinguish spare parts from consumables. If uncertain, add '(uncertain)' suffix."
|
||||
|
||||
4. Do NOT change any existing extraction logic or output structure. Only ADD the new section and clarify Category extraction.
|
||||
|
||||
**Code Quality:**
|
||||
- Preserve exact indentation and formatting from original prompt
|
||||
- Ensure multi-line string literals remain valid Python
|
||||
- No imports or logic changes — prompt enhancement only
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File gemini_extractor.py modified
|
||||
- Grep finds: "CLASSIFICATION GUIDE - SPARE PARTS vs CONSUMABLES" in the file
|
||||
- Grep finds: "Kingston Fury 16GB DDR4-3200" example in the file
|
||||
- Grep finds: "6ft SATA Cable" counter-example in the file
|
||||
- Grep finds: "Decision Tree:" decision logic in the file
|
||||
- Module still imports and initializes without errors: `python3 -c "from backend.ai.gemini_extractor import extract_item"`
|
||||
- Prompt structure remains intact (no broken f-strings or syntax errors)
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Enhance Claude AI Prompt with Spare-Parts Classification
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Mirror Gemini prompt enhancement in Claude extractor for consistent spare-parts classification across both AI providers.</objective>
|
||||
<read_first>
|
||||
- backend/ai/claude_extractor.py (current prompt structure and extraction pattern)
|
||||
- Task 2 output (what was added to Gemini prompt)
|
||||
- 4.1-RESEARCH.md section 3 (AI Prompt Enhancement)
|
||||
</read_first>
|
||||
<action>
|
||||
Modify file: backend/ai/claude_extractor.py
|
||||
|
||||
**Action Steps:**
|
||||
|
||||
1. Locate the extraction prompt in claude_extractor.py (similar structure to gemini_extractor.py)
|
||||
|
||||
2. Insert the SAME "CLASSIFICATION GUIDE - SPARE PARTS vs CONSUMABLES" section (from Task 2) after category/type extraction in Claude's prompt
|
||||
|
||||
3. Ensure Claude's Category field description includes the same guidance: "Use the Classification Guide above to distinguish spare parts from consumables. If uncertain, add '(uncertain)' suffix."
|
||||
|
||||
4. Preserve all existing Claude-specific instructions (model-specific prompt tuning, if any)
|
||||
|
||||
5. No logic changes — prompt enhancement only, identical content to Gemini
|
||||
|
||||
**Code Quality:**
|
||||
- Match the exact text from Gemini prompt (no divergence)
|
||||
- Preserve Claude's multi-turn or system prompt structure
|
||||
- Maintain compatibility with anthropic SDK
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File claude_extractor.py modified
|
||||
- Grep finds: "CLASSIFICATION GUIDE - SPARE PARTS vs CONSUMABLES" in the file
|
||||
- Grep finds: "Kingston Fury 16GB DDR4-3200" example in the file
|
||||
- Grep finds: "Decision Tree:" decision logic in the file
|
||||
- Module still imports and initializes without errors: `python3 -c "from backend.ai.claude_extractor import extract_item"`
|
||||
- Prompt structure remains intact (no broken f-strings or syntax errors)
|
||||
- Content is identical to Gemini prompt CLASSIFICATION GUIDE section
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Create Unit Tests for Spare-Parts Classification
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Write comprehensive pytest unit tests for spare-parts classification logic to validate accuracy and edge cases.</objective>
|
||||
<read_first>
|
||||
- backend/ai/spare_parts_whitelist.py (Task 1 output)
|
||||
- 4.1-RESEARCH.md section 5 (Testing & Validation Strategy) — specifically "Unit Tests" subsection
|
||||
- PROJECT_ARCHITECTURE.md section 2.1 (Testing: Pytest)
|
||||
</read_first>
|
||||
<action>
|
||||
Create file: tests/test_spare_parts_classification.py
|
||||
|
||||
**Test Structure (Pytest):**
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from backend.ai.spare_parts_whitelist import (
|
||||
classify_as_spare_part,
|
||||
is_consumable,
|
||||
get_spare_part_type
|
||||
)
|
||||
|
||||
class TestSparePartsClassification:
|
||||
"""Test spare-parts classification logic."""
|
||||
|
||||
def test_exact_match_ram(self):
|
||||
"""Exact match for RAM should return True."""
|
||||
assert classify_as_spare_part("RAM") is True
|
||||
assert classify_as_spare_part("DDR4") is True
|
||||
assert classify_as_spare_part("DRAM") is True
|
||||
|
||||
def test_exact_match_storage(self):
|
||||
"""Exact match for storage should return True."""
|
||||
assert classify_as_spare_part("SSD") is True
|
||||
assert classify_as_spare_part("NVME") is True
|
||||
assert classify_as_spare_part("HDD") is True
|
||||
|
||||
def test_exact_match_processor(self):
|
||||
"""Exact match for processors should return True."""
|
||||
assert classify_as_spare_part("CPU") is True
|
||||
assert classify_as_spare_part("GPU") is True
|
||||
assert classify_as_spare_part("PROCESSOR") is True
|
||||
|
||||
def test_exact_match_power(self):
|
||||
"""Exact match for power supplies should return True."""
|
||||
assert classify_as_spare_part("PSU") is True
|
||||
assert classify_as_spare_part("POWER SUPPLY UNIT") is True
|
||||
|
||||
def test_consumable_cables(self):
|
||||
"""Cables should return False."""
|
||||
assert classify_as_spare_part("6ft SATA Cable") is False
|
||||
assert classify_as_spare_part("USB Power Cable") is False
|
||||
assert classify_as_spare_part("Ethernet Cable") is False
|
||||
|
||||
def test_consumable_fasteners(self):
|
||||
"""Fasteners should return False."""
|
||||
assert classify_as_spare_part("CPU Mounting Hardware Kit") is False
|
||||
assert classify_as_spare_part("Screw Kit") is False
|
||||
assert classify_as_spare_part("Standoff Set") is False
|
||||
|
||||
def test_consumable_thermal_materials(self):
|
||||
"""Thermal materials should return False."""
|
||||
assert classify_as_spare_part("Thermal Paste") is False
|
||||
assert classify_as_spare_part("Thermal Pad") is False
|
||||
assert classify_as_spare_part("Adhesive Tape") is False
|
||||
|
||||
def test_fuzzy_match_ram(self):
|
||||
"""Fuzzy match for RAM variants should return True."""
|
||||
assert classify_as_spare_part("Random Access Memory") is True
|
||||
assert classify_as_spare_part("DDR 4") is True # with space
|
||||
assert classify_as_spare_part("ddr4") is True # lowercase
|
||||
|
||||
def test_fuzzy_match_storage(self):
|
||||
"""Fuzzy match for storage variants should return True."""
|
||||
assert classify_as_spare_part("Solid State Drive") is True
|
||||
assert classify_as_spare_part("NVMe Drive") is True
|
||||
|
||||
def test_case_insensitivity(self):
|
||||
"""Classification should be case-insensitive."""
|
||||
assert classify_as_spare_part("ram") is True
|
||||
assert classify_as_spare_part("SsD") is True
|
||||
assert classify_as_spare_part("sata cable") is False
|
||||
|
||||
def test_edge_case_power_cable_vs_psu(self):
|
||||
"""Power supply is spare part; power cable is consumable."""
|
||||
assert classify_as_spare_part("Corsair RM850x 850W Power Supply") is True
|
||||
assert classify_as_spare_part("6ft Power Cable AC Cord") is False
|
||||
|
||||
def test_is_consumable_function(self):
|
||||
"""is_consumable should be inverse of classify_as_spare_part."""
|
||||
assert is_consumable("RAM") is False
|
||||
assert is_consumable("SATA Cable") is True
|
||||
|
||||
def test_get_spare_part_type_returns_normalized_type(self):
|
||||
"""get_spare_part_type returns normalized category or None."""
|
||||
assert get_spare_part_type("DDR4 RAM") == "RAM"
|
||||
assert get_spare_part_type("Kingston SSD") == "SSD"
|
||||
assert get_spare_part_type("Intel CPU") == "CPU"
|
||||
assert get_spare_part_type("SATA Cable") is None
|
||||
```
|
||||
|
||||
**Execution & Verification:**
|
||||
- All tests must pass without errors
|
||||
- Minimum 12 test cases covering: exact match, fuzzy match, consumables, edge cases
|
||||
- Use pytest fixtures if needed for setup/teardown
|
||||
- No external API calls or network dependencies
|
||||
|
||||
**Code Quality:**
|
||||
- Use descriptive test names following pattern: `test_<feature>_<scenario>`
|
||||
- Include docstrings on each test method
|
||||
- Use assertions with clear expected values
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File exists: tests/test_spare_parts_classification.py
|
||||
- Test suite runs without errors: `pytest tests/test_spare_parts_classification.py -v`
|
||||
- Minimum 12 test cases implemented
|
||||
- Test passes: `test_exact_match_ram` and other exact match tests
|
||||
- Test passes: `test_consumable_cables` and other consumable tests
|
||||
- Test passes: `test_fuzzy_match_ram` for fuzzy matching
|
||||
- Test passes: `test_edge_case_power_cable_vs_psu` for edge cases
|
||||
- All assertions are positive (assert X is True/False, not assert not X)
|
||||
- Grep finds: "class TestSparePartsClassification:" in the file
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave 1 Summary
|
||||
|
||||
**What this wave accomplishes:**
|
||||
- Creates reusable spare-parts classification module with fuzzy matching (85-90% accuracy expected)
|
||||
- Enhances both Gemini and Claude prompts with consistent spare-parts decision tree
|
||||
- Provides comprehensive unit test coverage for classification logic
|
||||
- Establishes foundation for web search service in Wave 2
|
||||
|
||||
**Completion Criteria:**
|
||||
- All 4 tasks pass acceptance criteria
|
||||
- Pytest suite: `pytest tests/test_spare_parts_classification.py -v` → all tests pass
|
||||
- Code imports without errors:
|
||||
```bash
|
||||
python3 -c "from backend.ai.spare_parts_whitelist import classify_as_spare_part"
|
||||
python3 -c "from backend.ai.gemini_extractor import extract_item"
|
||||
python3 -c "from backend.ai.claude_extractor import extract_item"
|
||||
```
|
||||
- fuzzywuzzy dependency added to backend/requirements.txt
|
||||
|
||||
**Dependencies for Wave 2:**
|
||||
- spare_parts_whitelist.py module (Task 1)
|
||||
- Enhanced AI prompts (Tasks 2-3)
|
||||
- Classification tests passing (Task 4)
|
||||
|
||||
---
|
||||
670
.planning/phases/4.1-ai-spare-parts-deep-id/4.1-PLAN-02.md
Normal file
670
.planning/phases/4.1-ai-spare-parts-deep-id/4.1-PLAN-02.md
Normal file
@@ -0,0 +1,670 @@
|
||||
---
|
||||
wave: 2
|
||||
depends_on: ["4.1-PLAN-01.md"]
|
||||
files_modified:
|
||||
- path: "backend/services/spare_parts_search.py"
|
||||
- path: "backend/services/web_scraper.py"
|
||||
- path: "backend/services/spec_extractor.py"
|
||||
- path: "backend/routers/items.py"
|
||||
- path: "tests/test_spare_parts_search.py"
|
||||
- path: "backend/requirements.txt"
|
||||
autonomous: true
|
||||
---
|
||||
|
||||
# Phase 4.1 Wave 2: Web Scraping Service & Backend Integration
|
||||
|
||||
**Objective:** Implement web scraping and spec extraction services, integrate into `/api/onboarding/extract` endpoint, and add comprehensive backend tests.
|
||||
|
||||
**Prerequisites:** Wave 1 must be complete (spare_parts_whitelist.py, AI prompt enhancements, classification tests passing).
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Create Web Scraper Service
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Build HTTP request handler with rate limiting, User-Agent rotation, and fallback search engines (Google → Bing) for resilient spare-parts searching.</objective>
|
||||
<read_first>
|
||||
- 4.1-RESEARCH.md sections 1 (Web Scraping Best Practices) and 5 (Backend Integration — Rate Limiting Implementation)
|
||||
- PROJECT_ARCHITECTURE.md section 2.1 (Python 3.12+, FastAPI, async patterns)
|
||||
- No existing scraper — new file
|
||||
</read_first>
|
||||
<action>
|
||||
Create file: backend/services/web_scraper.py
|
||||
|
||||
**Module Structure:**
|
||||
|
||||
1. Import statements:
|
||||
```python
|
||||
import asyncio
|
||||
import random
|
||||
import time
|
||||
from typing import Optional, List
|
||||
import aiohttp
|
||||
from bs4 import BeautifulSoup
|
||||
```
|
||||
(Note: add aiohttp and beautifulsoup4 to requirements.txt)
|
||||
|
||||
2. Create constant: USER_AGENT_POOL (list of 10+ realistic User-Agent strings)
|
||||
```python
|
||||
USER_AGENT_POOL = [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (Chrome/120.0.0.0)",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (Firefox/121.0)",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (Safari/537.36)",
|
||||
# ... add 7+ more variations across Windows, Linux, macOS and Chrome/Firefox/Safari
|
||||
]
|
||||
```
|
||||
|
||||
3. Implement class: `SearchRateLimiter`
|
||||
- Method `__init__(self, requests_per_second: float = 0.2)` → (1 request per 5 seconds)
|
||||
- Method `async acquire(self)` → blocks until rate quota available, using token bucket algorithm
|
||||
- Attributes: `capacity`, `refill_rate`, `tokens`, `last_refill` (time-based)
|
||||
- Logic from 4.1-RESEARCH.md section 5 (Rate Limiting Implementation) pseudocode
|
||||
|
||||
4. Implement async function: `search_google(query: str, timeout: int = 10) -> Optional[List[dict]]`
|
||||
- Builds Google search URL: `f"https://www.google.com/search?q={urllib.parse.quote(query)}"`
|
||||
- Uses aiohttp with rotating User-Agent
|
||||
- Parses HTML with BeautifulSoup using CSS selector `div.g` (Google result container)
|
||||
- Returns list of dicts: `[{"title": str, "url": str, "snippet": str}, ...]` or None
|
||||
- On 429/403 error: log warning, return None
|
||||
- On timeout: raise asyncio.TimeoutError
|
||||
- Default timeout: 10 seconds
|
||||
|
||||
5. Implement async function: `search_bing(query: str, timeout: int = 10) -> Optional[List[dict]]`
|
||||
- Builds Bing search URL: `f"https://www.bing.com/search?q={urllib.parse.quote(query)}"`
|
||||
- Parses HTML with CSS selector `li.b_algo` (Bing result container)
|
||||
- Returns same format as search_google()
|
||||
- More stable than Google (less blocking)
|
||||
|
||||
6. Implement async function: `fetch_and_parse_html(url: str, timeout: int = 10) -> Optional[str]`
|
||||
- Fetches HTML from arbitrary URL
|
||||
- Returns HTML string or None on error
|
||||
- Timeout: 10 seconds default
|
||||
|
||||
**Code Quality:**
|
||||
- All functions are async (use `async def`)
|
||||
- Type hints on all parameters and returns
|
||||
- Docstrings with example usage
|
||||
- Exception handling: catch aiohttp.ClientError, asyncio.TimeoutError, and BeautifulSoup parse errors
|
||||
- No blocking I/O in async functions
|
||||
- Rate limiter uses time.time() for token bucket (not asyncio.sleep loops)
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File exists: backend/services/web_scraper.py
|
||||
- Grep finds: `class SearchRateLimiter:` in file
|
||||
- Grep finds: `async def search_google(` in file
|
||||
- Grep finds: `async def search_bing(` in file
|
||||
- Grep finds: `USER_AGENT_POOL` with 10+ entries
|
||||
- Module imports without error: `python3 -c "from backend.services.web_scraper import SearchRateLimiter, search_google, search_bing"`
|
||||
- Rate limiter has `acquire()` async method
|
||||
- Both search functions accept `query: str, timeout: int` parameters
|
||||
- Both search functions return `Optional[List[dict]]`
|
||||
- aiohttp and beautifulsoup4 added to backend/requirements.txt
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Create Spec Extractor Service
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Extract product specifications, manufacturer, model, and description from search results using regex patterns and data mapping to Item fields.</objective>
|
||||
<read_first>
|
||||
- 4.1-RESEARCH.md sections 4 (Search Result Parsing) with regex patterns and data extraction pipeline
|
||||
- 4.1-RESEARCH.md section 4 (Mapping to Item Fields) for spec extraction rules
|
||||
- PROJECT_ARCHITECTURE.md section 3 (Item model fields: Category, Type, Notes)
|
||||
</read_first>
|
||||
<action>
|
||||
Create file: backend/services/spec_extractor.py
|
||||
|
||||
**Module Structure:**
|
||||
|
||||
1. Import statements:
|
||||
```python
|
||||
import re
|
||||
from typing import Optional, Dict, Any
|
||||
from dataclasses import dataclass
|
||||
```
|
||||
|
||||
2. Create dataclass: `ExtractedSpecs`
|
||||
```python
|
||||
@dataclass
|
||||
class ExtractedSpecs:
|
||||
manufacturer: Optional[str]
|
||||
model: Optional[str]
|
||||
capacity: Optional[str] # e.g., "16GB"
|
||||
memory_type: Optional[str] # e.g., "DDR4"
|
||||
speed: Optional[str] # e.g., "3200MHz"
|
||||
latency: Optional[str] # e.g., "CAS 16"
|
||||
storage_type: Optional[str] # e.g., "SSD", "HDD"
|
||||
processor_brand: Optional[str] # e.g., "Intel"
|
||||
processor_model: Optional[str] # e.g., "Core i7-12700K"
|
||||
power_rating: Optional[str] # e.g., "850W"
|
||||
description: str # Full snippet/details from search
|
||||
confidence: float # 0.0-1.0 score
|
||||
|
||||
def to_item_fields(self, category: str) -> Dict[str, str]:
|
||||
"""Map extracted specs to Item model fields."""
|
||||
# Implementation: see action below
|
||||
```
|
||||
|
||||
3. Implement function: `extract_specs_from_snippet(snippet: str, title: str, category: str) -> ExtractedSpecs`
|
||||
- Input: search result title, snippet, and item category
|
||||
- Uses regex patterns from 4.1-RESEARCH.md section 4:
|
||||
- Memory: `r'\b(\d+)\s*(GB|TB)\s*(DDR\d|DRAM|RAM|SDRAM)'`
|
||||
- Storage: `r'\b(\d+)\s*(GB|TB)\s*(SSD|NVME|NVMe|M\.2|HDD|SATA)'`
|
||||
- Processor: `r'(Intel|AMD)\s+([A-Z0-9-]+)\s*(\d+\.\d+\s*GHz)?'`
|
||||
- Power: `r'\b(\d+)\s*(W|watts?|watt)\s*(power|supply|PSU)'`
|
||||
- Speed/Latency: `r'(\d+)\s*(MHz|GHz|CAS|Latency)'`
|
||||
- Extract manufacturer: check title/snippet for known brands (Kingston, Samsung, Intel, AMD, Corsair, etc.)
|
||||
- Build confidence score:
|
||||
- Exact part match in snippet: +0.2
|
||||
- All major specs found: +0.3
|
||||
- Manufacturer + model: +0.2
|
||||
- Consistency checks (e.g., DDR4 with GHz speed): +0.25
|
||||
- Return ExtractedSpecs dataclass
|
||||
|
||||
4. Implement method: `ExtractedSpecs.to_item_fields(category: str) -> Dict[str, str]`
|
||||
- Maps specs to Item model fields:
|
||||
- **Item.Category**: category (from whitelist)
|
||||
- **Item.Type**: formatted as "[manufacturer] [model] [capacity/speed]" or specific type (e.g., "DDR4", "NVMe")
|
||||
- **Item.Notes**: full description including all extracted specs
|
||||
- Example output:
|
||||
```python
|
||||
{
|
||||
"category": "RAM",
|
||||
"item_type": "Kingston Fury 16GB DDR4-3200",
|
||||
"notes": "Kingston Fury 16GB DDR4-3200MHz CAS Latency 16 - High-performance RAM module"
|
||||
}
|
||||
```
|
||||
|
||||
5. Implement function: `normalize_variations(text: str) -> str`
|
||||
- Normalizes common abbreviations:
|
||||
- "DDR4" ↔ "DDR 4" ↔ "DDR-4" → "DDR4"
|
||||
- "3200 MHz" ↔ "3200MHz" → "3200MHz"
|
||||
- "Intel i7" ↔ "Intel Core i7" → standardized format
|
||||
- Used in regex extraction for consistency
|
||||
|
||||
**Code Quality:**
|
||||
- All functions have type hints
|
||||
- Docstrings with example input/output
|
||||
- Regex patterns are compiled once as module constants (not in loop)
|
||||
- Error handling: gracefully handle missing fields (return None/default)
|
||||
- Confidence scoring is deterministic (no randomness)
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File exists: backend/services/spec_extractor.py
|
||||
- Grep finds: `class ExtractedSpecs:` in file
|
||||
- Grep finds: `def extract_specs_from_snippet(` in file
|
||||
- Grep finds: `def to_item_fields(` in file
|
||||
- Module imports without error: `python3 -c "from backend.services.spec_extractor import ExtractedSpecs, extract_specs_from_snippet"`
|
||||
- ExtractedSpecs dataclass has all fields: manufacturer, model, capacity, memory_type, speed, latency, storage_type, processor_brand, processor_model, power_rating, description, confidence
|
||||
- to_item_fields() returns Dict[str, str] with keys: category, item_type, notes
|
||||
- Example test: `extract_specs_from_snippet("Kingston Fury 16GB DDR4-3200 RAM", "...", "RAM")` returns ExtractedSpecs with confidence > 0.5
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Create Spare-Parts Search Orchestrator Service
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Build main orchestrator service that combines web scraping, rate limiting, and spec extraction with automatic fallback and timeout handling.</objective>
|
||||
<read_first>
|
||||
- 4.1-RESEARCH.md section 5 (Backend Integration Architecture) — Search Service Pseudocode
|
||||
- Task 1 output: backend/services/web_scraper.py
|
||||
- Task 2 output: backend/services/spec_extractor.py
|
||||
- backend/ai/spare_parts_whitelist.py (from Wave 1)
|
||||
</read_first>
|
||||
<action>
|
||||
Create file: backend/services/spare_parts_search.py
|
||||
|
||||
**Module Structure:**
|
||||
|
||||
1. Import statements:
|
||||
```python
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional
|
||||
from dataclasses import dataclass
|
||||
from backend.services.web_scraper import search_google, search_bing, SearchRateLimiter
|
||||
from backend.services.spec_extractor import ExtractedSpecs, extract_specs_from_snippet
|
||||
from backend.ai.spare_parts_whitelist import classify_as_spare_part
|
||||
```
|
||||
|
||||
2. Create dataclass: `SparePartSearchResult`
|
||||
```python
|
||||
@dataclass
|
||||
class SparePartSearchResult:
|
||||
status: str # "success", "timeout", "error", "no_results", "not_spare_part"
|
||||
specs: Optional[ExtractedSpecs] = None
|
||||
error: Optional[str] = None
|
||||
confidence: float = 0.0
|
||||
```
|
||||
|
||||
3. Create module-level rate limiter (singleton pattern):
|
||||
```python
|
||||
_rate_limiter = SearchRateLimiter(requests_per_second=0.2) # 1 request per 5 seconds
|
||||
```
|
||||
|
||||
4. Implement async function: `search_and_extract(part_number: str, category: str, manufacturer: Optional[str] = None, timeout: int = 20) -> SparePartSearchResult`
|
||||
- Algorithm (from 4.1-RESEARCH.md section 5):
|
||||
a. Check: is category in spare-parts whitelist? If not → return `SparePartSearchResult(status="not_spare_part", ...)`
|
||||
b. Build search query: `f"{part_number} {category} {manufacturer or ''}".strip()`
|
||||
c. Wrap in asyncio.timeout(timeout) block:
|
||||
- Acquire rate limiter: `await _rate_limiter.acquire()`
|
||||
- Try Google search: `results = await search_google(query)`
|
||||
- If no results → fallback to Bing: `results = await search_bing(query)`
|
||||
- If still no results → return `SparePartSearchResult(status="no_results", error="...")`
|
||||
- Parse best result (index 0): `specs = extract_specs_from_snippet(results[0]["title"], results[0]["snippet"], category)`
|
||||
- Return `SparePartSearchResult(status="success", specs=specs, confidence=specs.confidence)`
|
||||
d. On asyncio.TimeoutError → return `SparePartSearchResult(status="timeout", error="Search exceeded {timeout}s timeout")`
|
||||
e. On Exception → return `SparePartSearchResult(status="error", error=str(e))`
|
||||
|
||||
5. Implement logging:
|
||||
- Log all search attempts with query and category
|
||||
- Log timeouts, errors, and fallbacks (INFO level)
|
||||
- Log rate limiter waits (DEBUG level)
|
||||
- Use logger: `logging.getLogger(__name__)`
|
||||
|
||||
**Code Quality:**
|
||||
- All functions are async
|
||||
- Type hints on all parameters and returns
|
||||
- Docstrings with example usage
|
||||
- No external API calls to Google/Bing in unit tests (use mocks)
|
||||
- Graceful error handling for all network failures
|
||||
- Timeout is enforced by asyncio.timeout() context manager (exact timeout from parameter)
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File exists: backend/services/spare_parts_search.py
|
||||
- Grep finds: `class SparePartSearchResult:` in file
|
||||
- Grep finds: `async def search_and_extract(` in file
|
||||
- Module imports without error: `python3 -c "from backend.services.spare_parts_search import search_and_extract, SparePartSearchResult"`
|
||||
- SparePartSearchResult has fields: status, specs, error, confidence
|
||||
- search_and_extract accepts parameters: part_number: str, category: str, manufacturer: Optional[str], timeout: int
|
||||
- Function returns SparePartSearchResult with appropriate status values
|
||||
- Rate limiter is module-level singleton: `_rate_limiter = SearchRateLimiter(...)`
|
||||
- Timeout is enforced via asyncio.timeout() context manager
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Integrate Search into `/api/onboarding/extract` Endpoint
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Modify backend router to call spare-parts search service after AI extraction when category matches whitelist and part number exists.</objective>
|
||||
<read_first>
|
||||
- backend/routers/items.py (locate `/api/onboarding/extract` endpoint)
|
||||
- 4.1-CONTEXT.md decisions D-05, D-06, D-07, D-08 (search trigger and user flow)
|
||||
- 4.1-RESEARCH.md section 5 (Integration Flow in `/api/onboarding/extract`)
|
||||
- Tasks 1-3 output (all search services)
|
||||
</read_first>
|
||||
<action>
|
||||
Modify file: backend/routers/items.py
|
||||
|
||||
**Action Steps:**
|
||||
|
||||
1. Add imports at top of file:
|
||||
```python
|
||||
from backend.services.spare_parts_search import search_and_extract as search_spare_parts
|
||||
from backend.ai.spare_parts_whitelist import classify_as_spare_part
|
||||
import asyncio
|
||||
```
|
||||
|
||||
2. Locate the `/api/onboarding/extract` POST endpoint (should return extracted item data from AI)
|
||||
|
||||
3. Modify endpoint logic AFTER AI extraction step (Gemini or Claude):
|
||||
```python
|
||||
# Existing AI extraction code...
|
||||
ai_data = await extract_with_gemini_or_claude(...) # Returns: {name, category, item_type, part_number, ...}
|
||||
|
||||
# NEW: Check if search should be triggered
|
||||
search_results = None
|
||||
search_status = "skipped"
|
||||
search_error = None
|
||||
|
||||
category = ai_data.get("category", "").strip()
|
||||
part_number = ai_data.get("part_number", "").strip()
|
||||
|
||||
if classify_as_spare_part(category) and part_number:
|
||||
# Trigger spare-parts search
|
||||
try:
|
||||
manufacturer = ai_data.get("manufacturer", "")
|
||||
search_result = await search_spare_parts(
|
||||
part_number=part_number,
|
||||
category=category,
|
||||
manufacturer=manufacturer,
|
||||
timeout=20 # 20-30 seconds from RESEARCH.md
|
||||
)
|
||||
|
||||
search_status = search_result.status
|
||||
search_error = search_result.error
|
||||
|
||||
if search_result.status == "success" and search_result.specs:
|
||||
search_results = search_result.specs.to_item_fields(category)
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
search_status = "timeout"
|
||||
search_error = "Search exceeded 20 second timeout"
|
||||
except Exception as e:
|
||||
search_status = "error"
|
||||
search_error = str(e)
|
||||
|
||||
# Return combined response
|
||||
return {
|
||||
"ai_data": ai_data,
|
||||
"search_results": search_results,
|
||||
"search_status": search_status,
|
||||
"search_error": search_error
|
||||
}
|
||||
```
|
||||
|
||||
4. Response schema should include:
|
||||
- `ai_data`: dict with original AI-extracted fields
|
||||
- `search_results`: dict with `{category, item_type, notes}` or null if skipped/failed
|
||||
- `search_status`: string enum ["success", "timeout", "error", "no_results", "skipped", "not_spare_part"]
|
||||
- `search_error`: error message string or null
|
||||
|
||||
5. Ensure endpoint remains async and doesn't block other requests
|
||||
|
||||
**Code Quality:**
|
||||
- No changes to existing AI extraction logic
|
||||
- Search is called conditionally (only if category matches AND part_number exists)
|
||||
- Timeout is enforced (20 seconds from RESEARCH.md)
|
||||
- Errors are caught and returned in response (not raising exceptions)
|
||||
- Response structure matches frontend expectations (from RESEARCH.md section 6)
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File backend/routers/items.py modified
|
||||
- Grep finds: `from backend.services.spare_parts_search import search_spare_parts` in file
|
||||
- Grep finds: `from backend.ai.spare_parts_whitelist import classify_as_spare_part` in file
|
||||
- Grep finds: `classify_as_spare_part(category) and part_number:` in file
|
||||
- Grep finds: `search_status = search_result.status` in file
|
||||
- Endpoint returns dict with keys: ai_data, search_results, search_status, search_error
|
||||
- Endpoint is still async function (no blocking calls)
|
||||
- Timeout is set to 20 seconds: `timeout=20`
|
||||
- Search is conditional: only triggered if category is spare part AND part_number exists
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Create Backend Tests for Search Services
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Write comprehensive pytest tests for search orchestrator, web scraper, and spec extractor with mocked HTTP responses.</objective>
|
||||
<read_first>
|
||||
- 4.1-RESEARCH.md section 8 (Testing & Validation Strategy) — Unit Tests and Integration Tests subsections
|
||||
- Tasks 1-4 output (all services)
|
||||
- PROJECT_ARCHITECTURE.md section 2.1 (Testing: Pytest)
|
||||
</read_first>
|
||||
<action>
|
||||
Create file: tests/test_spare_parts_search.py
|
||||
|
||||
**Test Structure (Pytest with pytest-asyncio for async tests):**
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
from backend.services.spare_parts_search import search_and_extract, SparePartSearchResult
|
||||
from backend.services.spec_extractor import extract_specs_from_snippet
|
||||
from backend.ai.spare_parts_whitelist import classify_as_spare_part
|
||||
|
||||
class TestSparePartsSearch:
|
||||
"""Test spare-parts search orchestrator."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_and_extract_not_spare_part(self):
|
||||
"""Non-spare-parts category should skip search."""
|
||||
result = await search_and_extract(
|
||||
part_number="6ft Cable",
|
||||
category="Cable",
|
||||
timeout=20
|
||||
)
|
||||
assert result.status == "not_spare_part"
|
||||
assert result.specs is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_and_extract_no_part_number(self):
|
||||
"""Missing part number should skip search."""
|
||||
result = await search_and_extract(
|
||||
part_number="",
|
||||
category="RAM",
|
||||
timeout=20
|
||||
)
|
||||
assert result.status == "skipped"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch('backend.services.spare_parts_search.search_google')
|
||||
async def test_search_and_extract_success(self, mock_google):
|
||||
"""Successful search should return specs."""
|
||||
mock_google.return_value = [
|
||||
{
|
||||
"title": "Kingston Fury 16GB DDR4-3200",
|
||||
"snippet": "Kingston Fury 16GB DDR4-3200MHz CAS Latency 16",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
]
|
||||
|
||||
result = await search_and_extract(
|
||||
part_number="Kingston Fury 16GB",
|
||||
category="RAM",
|
||||
timeout=20
|
||||
)
|
||||
|
||||
assert result.status == "success"
|
||||
assert result.specs is not None
|
||||
assert result.confidence > 0.5
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_and_extract_timeout(self):
|
||||
"""Timeout should return timeout status."""
|
||||
result = await search_and_extract(
|
||||
part_number="Kingston Fury 16GB",
|
||||
category="RAM",
|
||||
timeout=0.001 # Force timeout
|
||||
)
|
||||
assert result.status == "timeout"
|
||||
assert result.specs is None
|
||||
assert "timeout" in result.error.lower()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch('backend.services.spare_parts_search.search_google')
|
||||
@patch('backend.services.spare_parts_search.search_bing')
|
||||
async def test_search_fallback_to_bing(self, mock_bing, mock_google):
|
||||
"""Should fallback to Bing if Google returns no results."""
|
||||
mock_google.return_value = None
|
||||
mock_bing.return_value = [
|
||||
{
|
||||
"title": "Samsung 970 EVO 1TB NVMe",
|
||||
"snippet": "Samsung 970 EVO 1TB NVMe SSD",
|
||||
"url": "https://example.com"
|
||||
}
|
||||
]
|
||||
|
||||
result = await search_and_extract(
|
||||
part_number="Samsung 970 EVO",
|
||||
category="SSD",
|
||||
timeout=20
|
||||
)
|
||||
|
||||
assert result.status == "success"
|
||||
mock_bing.assert_called_once()
|
||||
|
||||
class TestSpecExtractor:
|
||||
"""Test specification extraction from search results."""
|
||||
|
||||
def test_extract_specs_from_snippet_ram(self):
|
||||
"""Extract RAM specifications."""
|
||||
specs = extract_specs_from_snippet(
|
||||
snippet="Kingston Fury 16GB DDR4-3200MHz CAS Latency 16",
|
||||
title="Kingston Fury 16GB DDR4-3200",
|
||||
category="RAM"
|
||||
)
|
||||
|
||||
assert specs.manufacturer == "Kingston"
|
||||
assert specs.capacity == "16GB"
|
||||
assert specs.memory_type == "DDR4"
|
||||
assert specs.speed == "3200"
|
||||
|
||||
def test_extract_specs_from_snippet_ssd(self):
|
||||
"""Extract SSD specifications."""
|
||||
specs = extract_specs_from_snippet(
|
||||
snippet="Samsung 970 EVO 1TB NVMe SSD",
|
||||
title="Samsung 970 EVO 1TB",
|
||||
category="SSD"
|
||||
)
|
||||
|
||||
assert specs.manufacturer == "Samsung"
|
||||
assert specs.capacity == "1TB"
|
||||
assert specs.storage_type == "NVMe"
|
||||
|
||||
def test_to_item_fields_mapping(self):
|
||||
"""Test mapping specs to Item model fields."""
|
||||
specs = extract_specs_from_snippet(
|
||||
snippet="Kingston Fury 16GB DDR4-3200MHz",
|
||||
title="Kingston Fury 16GB DDR4-3200",
|
||||
category="RAM"
|
||||
)
|
||||
|
||||
item_fields = specs.to_item_fields("RAM")
|
||||
|
||||
assert item_fields["category"] == "RAM"
|
||||
assert "Kingston" in item_fields["item_type"]
|
||||
assert "16GB" in item_fields["notes"]
|
||||
|
||||
class TestWhitelistIntegration:
|
||||
"""Test whitelist integration with search."""
|
||||
|
||||
def test_classify_spare_part_enables_search(self):
|
||||
"""Spare parts should enable search."""
|
||||
assert classify_as_spare_part("RAM") is True
|
||||
assert classify_as_spare_part("SSD") is True
|
||||
|
||||
def test_consumable_disables_search(self):
|
||||
"""Consumables should skip search."""
|
||||
assert classify_as_spare_part("Cable") is False
|
||||
assert classify_as_spare_part("Thermal Paste") is False
|
||||
```
|
||||
|
||||
**Test Execution:**
|
||||
- All tests must pass: `pytest tests/test_spare_parts_search.py -v`
|
||||
- Async tests use `@pytest.mark.asyncio` decorator
|
||||
- Mock external HTTP calls (don't make real requests to Google/Bing)
|
||||
- Use `pytest-asyncio` package for async support
|
||||
|
||||
**Code Quality:**
|
||||
- Descriptive test names
|
||||
- Docstrings on each test
|
||||
- Clear assertions with expected values
|
||||
- Minimum 10 test cases
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File exists: tests/test_spare_parts_search.py
|
||||
- Test suite runs without errors: `pytest tests/test_spare_parts_search.py -v`
|
||||
- Minimum 10 test cases implemented
|
||||
- Test passes: `test_search_and_extract_not_spare_part`
|
||||
- Test passes: `test_search_and_extract_timeout`
|
||||
- Test passes: `test_search_fallback_to_bing` (using mocked Bing)
|
||||
- Test passes: `test_extract_specs_from_snippet_ram`
|
||||
- Test passes: `test_to_item_fields_mapping`
|
||||
- pytest-asyncio added to backend/requirements.txt
|
||||
- All external HTTP calls are mocked (no real requests in tests)
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Update Backend Dependencies
|
||||
|
||||
```xml
|
||||
<task>
|
||||
<objective>Add new Python packages to requirements.txt with version constraints for all services created in Wave 2.</objective>
|
||||
<read_first>
|
||||
- backend/requirements.txt (current state)
|
||||
- AI_RULES.md section 2 (DEPENDENCIES: Update requirements.txt with version constraints)
|
||||
- Tasks 1-5 (all new services)
|
||||
</read_first>
|
||||
<action>
|
||||
Modify file: backend/requirements.txt
|
||||
|
||||
**Action Steps:**
|
||||
|
||||
1. Add these lines (in alphabetical order if file is sorted):
|
||||
```
|
||||
aiohttp==3.9.1
|
||||
beautifulsoup4==4.12.2
|
||||
fuzzywuzzy==0.18.0
|
||||
python-Levenshtein==0.21.1
|
||||
pytest-asyncio==0.23.2
|
||||
```
|
||||
|
||||
2. Verify no duplicate entries exist in file
|
||||
|
||||
3. Ensure all existing dependencies remain unchanged (only ADD new ones)
|
||||
|
||||
**Rationale:**
|
||||
- **aiohttp**: Async HTTP client for web scraping in web_scraper.py
|
||||
- **beautifulsoup4**: HTML parsing for search results
|
||||
- **fuzzywuzzy**: Fuzzy string matching for spare-parts classification (added in Wave 1)
|
||||
- **python-Levenshtein**: Fast Levenshtein distance for fuzzywuzzy
|
||||
- **pytest-asyncio**: Async test support for pytest
|
||||
|
||||
**Code Quality:**
|
||||
- Use specific version pinning (major.minor.patch) for stability
|
||||
- No pre-release versions (no alpha/beta)
|
||||
- Versions chosen from stable releases as of 2026-04
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- File backend/requirements.txt modified
|
||||
- Grep finds: `aiohttp==3.9.1` in file
|
||||
- Grep finds: `beautifulsoup4==4.12.2` in file
|
||||
- Grep finds: `fuzzywuzzy==0.18.0` in file
|
||||
- Grep finds: `pytest-asyncio==0.23.2` in file
|
||||
- No duplicate entries in file
|
||||
- All existing dependencies remain unchanged
|
||||
- File has no syntax errors (can run `pip install -r backend/requirements.txt` without parsing errors)
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave 2 Summary
|
||||
|
||||
**What this wave accomplishes:**
|
||||
- Creates resilient web scraping service with fallback engines and rate limiting
|
||||
- Builds spec extraction service with regex patterns and confidence scoring
|
||||
- Implements orchestrator service combining all search logic with timeout handling
|
||||
- Integrates search into backend API endpoint for automatic spare-parts lookup
|
||||
- Provides comprehensive backend tests with mocked HTTP
|
||||
|
||||
**Completion Criteria:**
|
||||
- All 6 tasks pass acceptance criteria
|
||||
- Backend tests pass: `pytest tests/test_spare_parts_search.py -v` → all tests pass
|
||||
- All services import without error:
|
||||
```bash
|
||||
python3 -c "from backend.services.web_scraper import search_google, search_bing"
|
||||
python3 -c "from backend.services.spec_extractor import extract_specs_from_snippet"
|
||||
python3 -c "from backend.services.spare_parts_search import search_and_extract"
|
||||
```
|
||||
- `/api/onboarding/extract` endpoint returns search results in expected format
|
||||
- Dependencies installed: `pip install -r backend/requirements.txt` → no errors
|
||||
|
||||
**Dependencies for Wave 3:**
|
||||
- All search services (Tasks 1-3)
|
||||
- Backend API integration (Task 4)
|
||||
- Backend tests passing (Task 5)
|
||||
- Dependencies installed (Task 6)
|
||||
|
||||
---
|
||||
1142
.planning/phases/4.1-ai-spare-parts-deep-id/4.1-PLAN-03.md
Normal file
1142
.planning/phases/4.1-ai-spare-parts-deep-id/4.1-PLAN-03.md
Normal file
File diff suppressed because it is too large
Load Diff
761
.planning/phases/4.1-ai-spare-parts-deep-id/4.1-RESEARCH.md
Normal file
761
.planning/phases/4.1-ai-spare-parts-deep-id/4.1-RESEARCH.md
Normal file
@@ -0,0 +1,761 @@
|
||||
# Phase 4.1 Research: AI Prompt Enhancement — Spare Parts Deep Identification
|
||||
|
||||
**Research Date:** 2026-04-22
|
||||
**Scope:** Web scraping implementation, spare-parts classification, AI prompt enhancement, search result parsing, backend/frontend integration, and performance/scalability.
|
||||
|
||||
---
|
||||
|
||||
## 1. Web Scraping Best Practices: Python Requests + BeautifulSoup
|
||||
|
||||
### Key Findings
|
||||
|
||||
**Approach & Risks:**
|
||||
- **Direct Google scraping** is technically feasible but risky: Google actively detects and blocks scrapers with 429 (Too Many Requests) errors, CAPTCHA challenges, and IP bans.
|
||||
- **Terms of Service violation**: Google's ToS explicitly forbids scraping search results.
|
||||
- **HTML structure volatility**: Google changes CSS selectors and HTML markup frequently, breaking scrapers.
|
||||
- **Practical reality**: Direct scraping works for low-volume scenarios (tens of requests/hour) with proper mitigations.
|
||||
|
||||
**Safer Alternatives:**
|
||||
1. **SerpAPI / Similar APIs**: Officially maintained, handles blocking/rotation, but costs money ($5-50/month depending on volume).
|
||||
2. **Bing scraping**: Less aggressively blocked than Google, similar HTML structure, viable fallback.
|
||||
3. **Manufacturer sites** (Dell, HP, Kingston, Crucial): Most reliable source for spare-part specs.
|
||||
4. **GitHub Issues / StackOverflow**: Often contain real-world component usage and specifications.
|
||||
|
||||
**Recommended Hybrid Approach:**
|
||||
- Primary: Search manufacturer specs directly (most accurate).
|
||||
- Fallback 1: Bing web search with BeautifulSoup.
|
||||
- Fallback 2: Google search (if Bing returns no results).
|
||||
- Fallback 3: Return AI-extracted data only (graceful offline degradation).
|
||||
|
||||
### Rate Limiting Strategies
|
||||
|
||||
**Implementation:**
|
||||
- **Delay between requests**: 2-5 seconds minimum (random jitter recommended).
|
||||
- **User-Agent rotation**: Cycle through 10+ realistic User-Agent strings (Chrome, Firefox, Safari across Windows/Mac/Linux).
|
||||
- **Exponential backoff**: 1s → 2s → 4s → 8s → fail.
|
||||
- **Token bucket algorithm**: Max 0.2 requests/second (1 request per 5 seconds) per IP.
|
||||
|
||||
**User-Agent Pool (Examples):**
|
||||
```
|
||||
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (Chrome/120.0.0.0)
|
||||
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (Firefox/121.0)
|
||||
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (Safari/537.36)
|
||||
```
|
||||
|
||||
### Error Handling & Timeout Strategies
|
||||
|
||||
**HTTP Status Codes:**
|
||||
- **429 (Too Many Requests)**: Wait 10 seconds, retry once, then fail gracefully.
|
||||
- **403 (Forbidden)**: IP blocked; rotate User-Agent, increase delay, or skip.
|
||||
- **500+ (Server Error)**: Retry with exponential backoff.
|
||||
- **Timeout (>10s)**: Abort search, return AI data only, log warning.
|
||||
|
||||
**CAPTCHA Detection:**
|
||||
- BeautifulSoup can detect CAPTCHA forms by checking for `<form>` with `recaptcha` keywords.
|
||||
- If detected: Abort search immediately, return AI data, log incident.
|
||||
|
||||
**Latency Profile:**
|
||||
- Typical Google request: 2-8 seconds.
|
||||
- BeautifulSoup HTML parsing: 100-500ms.
|
||||
- Regex spec extraction: 10-50ms.
|
||||
- **Total end-to-end: 3-15 seconds (up to 30s with retries).**
|
||||
|
||||
---
|
||||
|
||||
## 2. Spare-Parts Classification Strategy
|
||||
|
||||
### Comprehensive Whitelist
|
||||
|
||||
**Spare-Part Categories (Include These):**
|
||||
- **Memory**: RAM, DRAM, DDR3, DDR4, DDR5, SODIMM, DIMM
|
||||
- **Storage**: SSD, NVME, M.2, SATA, HDD, hard drive, solid state drive
|
||||
- **Processors**: CPU, processor, APU, GPU, graphics card, discrete GPU
|
||||
- **Power**: PSU (power supply unit), adapter, power module (NOT cables/cords)
|
||||
- **Expansion Cards**: PCIe, PCI, RAID controller, network card (NIC), graphics card
|
||||
- **Cooling**: Heatsink, CPU cooler, thermal solution
|
||||
- **Motherboards**: Motherboard, BIOS, chipset
|
||||
|
||||
**Consumables to Exclude:**
|
||||
- Cables: SATA cables, USB cables, Ethernet cables, power cords.
|
||||
- Fasteners: Screws, washers, bolts, standoffs.
|
||||
- Adhesives/Thermal Materials: Thermal paste, thermal pads, adhesive tapes.
|
||||
- Connectors: Plugs, sockets, adapters (unless branded components).
|
||||
|
||||
**Edge Case: Power Supplies**
|
||||
- **Spare part**: "Corsair RM850x 850W Power Supply Unit" (replaceable, has specs).
|
||||
- **Consumable**: "6ft Power Cable" or "AC Power Cord" (generic utility item).
|
||||
|
||||
### Fuzzy Matching Implementation
|
||||
|
||||
**Strategy:**
|
||||
1. **Exact keyword match** (highest priority): Check if extracted Category contains exact whitelist terms (RAM, SSD, CPU, GPU, PSU).
|
||||
2. **Fuzzy matching** (Levenshtein distance, 70-80% threshold):
|
||||
- "Random Access Memory" → matches "RAM"
|
||||
- "Solid State Disk" → matches "SSD"
|
||||
3. **Regex patterns** (fallback):
|
||||
- `\bRAM\b|\bDRAM\b|\bDDR\d\b` → Memory component.
|
||||
- `\bSSD\b|\bNVME\b|\bM\.2\b` → Storage component.
|
||||
4. **Exclusion patterns** (reject consumables):
|
||||
- `^(cable|cord|fastener|screw|adhesive|thermal paste)$` (case-insensitive).
|
||||
|
||||
**Scoring System:**
|
||||
- Exact match in whitelist: +100 points → **Spare Part**.
|
||||
- Fuzzy match >80%: +50 points.
|
||||
- Fuzzy match 70-80%: +30 points.
|
||||
- Found in consumable exclusion list: -100 points → **Consumable**.
|
||||
- **Threshold**: Score ≥ 40 → Spare Part; < 40 → Unknown/Consumable.
|
||||
|
||||
---
|
||||
|
||||
## 3. AI Prompt Enhancement: Gemini 2.0 Flash & Claude 3.5 Sonnet
|
||||
|
||||
### Current State
|
||||
- **Gemini prompt**: Located in `backend/ai/prompts/gemini_extraction_prompt.md`.
|
||||
- **Claude prompt**: Located in `backend/ai/prompts/claude_extraction_prompt.md`.
|
||||
- Both focus on OCR extraction from label images.
|
||||
|
||||
### Phase 4.1 Enhancements
|
||||
|
||||
**New Classification Logic to Add:**
|
||||
|
||||
Insert into both prompts a new section after Category extraction:
|
||||
|
||||
```
|
||||
CLASSIFICATION GUIDE - SPARE PARTS vs CONSUMABLES:
|
||||
|
||||
Spare Parts (replaceable components that plug into or interface with devices):
|
||||
- RAM, DDR memory modules
|
||||
- SSDs, NVMe drives, M.2 modules
|
||||
- CPUs, GPUs, processors
|
||||
- Power supply units (PSU), power modules
|
||||
- Expansion cards (PCIe, RAID, NIC)
|
||||
- Cooling solutions (heatsinks, coolers)
|
||||
- Motherboards
|
||||
|
||||
NOT Spare Parts (consumables, generic items):
|
||||
- Cables (power, SATA, USB, Ethernet)
|
||||
- Fasteners (screws, washers, standoffs)
|
||||
- Thermal paste, thermal pads, adhesives
|
||||
- Connectors, plugs, sockets
|
||||
- Generic cords and adapters
|
||||
|
||||
Decision Tree:
|
||||
1. Does the item have a replaceable function in a larger system?
|
||||
2. Does it have a manufacturer part number and technical specifications?
|
||||
3. Is it described with model/revision information?
|
||||
If YES to 2+ questions: SPARE PART
|
||||
If item matches consumable examples: CONSUMABLE
|
||||
Otherwise: Mark as "uncertain" for human review.
|
||||
|
||||
Examples:
|
||||
✓ "Kingston Fury 16GB DDR4-3200" → Spare Part (RAM)
|
||||
✓ "Samsung 970 EVO 1TB NVMe" → Spare Part (SSD)
|
||||
✓ "Intel Core i7-12700K" → Spare Part (CPU)
|
||||
✗ "6ft SATA Cable" → Consumable (cable)
|
||||
✗ "CPU Mounting Hardware Kit" → Consumable (fasteners)
|
||||
```
|
||||
|
||||
### Testing Approach for Prompt Accuracy
|
||||
|
||||
**Validation Dataset:**
|
||||
1. Create 20-30 labeled images of actual spare-parts and consumables.
|
||||
2. Test both Gemini and Claude on same dataset.
|
||||
3. Measure accuracy of Category classification.
|
||||
4. Measure accuracy of Part Number extraction.
|
||||
5. Iterate on prompt examples until >95% accuracy on test set.
|
||||
|
||||
**Field Testing with Users:**
|
||||
1. Have 3-5 field users test Phase 4.1 with real items.
|
||||
2. Collect feedback on search quality and auto-population accuracy.
|
||||
3. Measure time-to-save improvement (before vs. after search integration).
|
||||
|
||||
---
|
||||
|
||||
## 4. Search Result Parsing: CSS Selectors & Data Extraction
|
||||
|
||||
### CSS Selectors for Google Search Results
|
||||
|
||||
**Standard HTML structure (may change):**
|
||||
```
|
||||
div.g // Result container
|
||||
├── h3 (or a[data-sokoban-click]) // Title
|
||||
├── a[href^='http'] // URL link
|
||||
├── div.VwiC3b (or similar) // Snippet/description
|
||||
└── div.eFM0qc // Display URL
|
||||
```
|
||||
|
||||
**Bing Search Selectors (more stable):**
|
||||
```
|
||||
li.b_algo // Result container
|
||||
├── h2 a // Title + link
|
||||
├── p // Snippet
|
||||
└── .tMee // Display URL
|
||||
```
|
||||
|
||||
### Spec Extraction from Snippets
|
||||
|
||||
**Regex Patterns:**
|
||||
|
||||
```python
|
||||
# Memory
|
||||
r'\b(\d+)\s*(GB|TB)\s*(DDR\d|DRAM|RAM|SDRAM)'
|
||||
# Storage
|
||||
r'\b(\d+)\s*(GB|TB)\s*(SSD|NVME|NVMe|M\.2|HDD|SATA)'
|
||||
# Processor
|
||||
r'(Intel|AMD)\s+([A-Z0-9-]+)\s*(\d+\.\d+\s*GHz)?'
|
||||
# Power
|
||||
r'\b(\d+)\s*(W|watts?|watt)\s*(power|supply|PSU)'
|
||||
# Speed/Latency
|
||||
r'(\d+)\s*(MHz|GHz|CAS|Latency)'
|
||||
```
|
||||
|
||||
### Data Extraction Pipeline
|
||||
|
||||
**Example Input:**
|
||||
```
|
||||
Title: Kingston Fury 16GB DDR4-3200 RAM Memory Module
|
||||
Snippet: Kingston Fury 16GB DDR4 3200MHz CAS Latency 16 - Get superior performance
|
||||
with Kingston FURY DDR4 memory. 16GB modules deliver rock-solid stability...
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
{
|
||||
"manufacturer": "Kingston",
|
||||
"model": "Fury",
|
||||
"capacity": "16GB",
|
||||
"memory_type": "DDR4",
|
||||
"speed": "3200MHz",
|
||||
"latency": "CAS 16",
|
||||
"confidence": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
**Mapping to Item Fields:**
|
||||
- `Item.Name`: `[Kingston] [Fury] [16GB] [DDR4-3200]` (cleaned)
|
||||
- `Item.Category`: "RAM" (from whitelist match)
|
||||
- `Item.Type`: "Memory Module" or "DDR4" (spareable)
|
||||
- `Item.Notes`: Full specs: "Kingston Fury 16GB DDR4-3200MHz CAS Latency 16"
|
||||
|
||||
### Handling Variations & Abbreviations
|
||||
|
||||
**Common variations to normalize:**
|
||||
- "DDR4" ↔ "DDR 4" ↔ "DDR-4"
|
||||
- "3200 MHz" ↔ "3200MHz" ↔ "3.2 GHz"
|
||||
- "Intel i7" ↔ "Intel Core i7" ↔ "Intel Core™ i7"
|
||||
- Manufacturers: "SK Hynix" ↔ "SK Hynix" (normalize spacing)
|
||||
|
||||
**Confidence scoring:**
|
||||
- Exact part number match: +0.2
|
||||
- All major specs found: +0.3
|
||||
- Manufacturer + model: +0.2
|
||||
- Consistency checks (price matches category): +0.25
|
||||
|
||||
---
|
||||
|
||||
## 5. Backend Integration Architecture
|
||||
|
||||
### New Modules to Create
|
||||
|
||||
1. **`backend/services/spare_parts_search.py`**
|
||||
- Main orchestrator service.
|
||||
- Public methods: `search_and_extract(part_number, category, timeout=20)`.
|
||||
- Returns: `SparePartSearchResult` dataclass.
|
||||
|
||||
2. **`backend/services/web_scraper.py`**
|
||||
- HTTP requests with User-Agent rotation and rate limiting.
|
||||
- Methods: `search_google()`, `search_bing()`, `fetch_and_parse_html()`.
|
||||
|
||||
3. **`backend/services/spec_extractor.py`**
|
||||
- Regex parsing and data extraction.
|
||||
- Methods: `extract_specs_from_snippet()`, `extract_specs_from_html()`.
|
||||
|
||||
4. **`backend/config/spare_parts_whitelist.py`**
|
||||
- Configurable category whitelist and exclusion patterns.
|
||||
- Easy to update without code changes.
|
||||
|
||||
### Integration Flow in `/api/onboarding/extract`
|
||||
|
||||
**Current Flow:**
|
||||
```
|
||||
1. User uploads image
|
||||
2. AI extraction (Gemini/Claude)
|
||||
3. Return extracted data to frontend
|
||||
```
|
||||
|
||||
**Phase 4.1 New Flow:**
|
||||
```
|
||||
1. User uploads image
|
||||
2. AI extraction (Gemini/Claude)
|
||||
3. Check: category in whitelist AND part_number exists?
|
||||
YES → Trigger async search
|
||||
NO → Return AI data, skip search
|
||||
4. Search executes (up to 30s timeout):
|
||||
- Try Google search
|
||||
- Fallback to Bing if Google fails
|
||||
- Parse results, extract specs
|
||||
5. Return: {
|
||||
ai_data: {...},
|
||||
search_results: {...} | null,
|
||||
search_status: "success" | "timeout" | "error" | "skipped",
|
||||
search_error: string | null
|
||||
}
|
||||
6. Frontend handles loading state, pre-populates fields
|
||||
```
|
||||
|
||||
### Search Service Pseudocode
|
||||
|
||||
```python
|
||||
async def search_and_extract(
|
||||
part_number: str,
|
||||
category: str,
|
||||
manufacturer: str | None = None,
|
||||
timeout: int = 20
|
||||
) -> SparePartSearchResult:
|
||||
"""
|
||||
Search for spare part specs and extract data.
|
||||
Returns immediately if timeout exceeded.
|
||||
"""
|
||||
try:
|
||||
# Build search query
|
||||
query = f"{part_number} {category} {manufacturer or ''}"
|
||||
|
||||
# Attempt search with timeout
|
||||
with asyncio.timeout(timeout):
|
||||
# Try Google first (with rate limiting)
|
||||
results = await search_google(query)
|
||||
|
||||
if not results:
|
||||
# Fallback to Bing
|
||||
results = await search_bing(query)
|
||||
|
||||
if not results:
|
||||
return SparePartSearchResult(
|
||||
status="no_results",
|
||||
specs=None,
|
||||
error="No search results found"
|
||||
)
|
||||
|
||||
# Parse best result
|
||||
specs = extract_specs_from_snippet(results[0])
|
||||
|
||||
return SparePartSearchResult(
|
||||
status="success",
|
||||
specs=specs,
|
||||
error=None,
|
||||
confidence=specs.get("confidence", 0.0)
|
||||
)
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
return SparePartSearchResult(
|
||||
status="timeout",
|
||||
specs=None,
|
||||
error="Search exceeded 20s timeout"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
return SparePartSearchResult(
|
||||
status="error",
|
||||
specs=None,
|
||||
error=str(e)
|
||||
)
|
||||
```
|
||||
|
||||
### Rate Limiting Implementation
|
||||
|
||||
**Token Bucket Algorithm:**
|
||||
```python
|
||||
class SearchRateLimiter:
|
||||
def __init__(self, requests_per_second: float = 0.2):
|
||||
# 0.2 req/sec = 1 req per 5 seconds
|
||||
self.capacity = 1.0
|
||||
self.refill_rate = requests_per_second
|
||||
self.tokens = 1.0
|
||||
self.last_refill = time.time()
|
||||
|
||||
async def acquire(self):
|
||||
"""Block until search quota available."""
|
||||
while self.tokens < 1.0:
|
||||
elapsed = time.time() - self.last_refill
|
||||
self.tokens += elapsed * self.refill_rate
|
||||
self.last_refill = time.time()
|
||||
if self.tokens < 1.0:
|
||||
await asyncio.sleep(0.1)
|
||||
self.tokens -= 1.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Frontend AIOnboarding Integration
|
||||
|
||||
### State Additions
|
||||
|
||||
```typescript
|
||||
interface AIOnboardingState {
|
||||
// ... existing state ...
|
||||
isSearching: boolean; // Search in progress
|
||||
searchError: string | null; // Error message if failed
|
||||
searchResults: SparePartSpecs | null; // Extracted specs
|
||||
searchTimeout: number; // Configurable timeout (30s default)
|
||||
}
|
||||
```
|
||||
|
||||
### UI Flow
|
||||
|
||||
**Sequence:**
|
||||
|
||||
1. **User confirms item** after AI extraction review.
|
||||
2. **Frontend calls** `POST /api/onboarding/extract` with image.
|
||||
3. **Backend returns** `{ai_data, search_results, search_status, search_error}`.
|
||||
4. **If search_status = "success"**:
|
||||
- Show `"Searching for specifications..."` modal (non-dismissible).
|
||||
- Spinner animation + countdown timer.
|
||||
- Pre-populate Item.Category, Item.Type, Item.Notes from search results.
|
||||
5. **User reviews all fields** (can edit any field).
|
||||
6. **User clicks Save** to commit to database.
|
||||
|
||||
**On Search Error:**
|
||||
- Show modal: `"Search failed: [error message]"`
|
||||
- Buttons: `[Retry Search] [Skip and Save]`
|
||||
- If Retry: Re-trigger search (max 2 retries).
|
||||
- If Skip: Use AI-extracted data only.
|
||||
|
||||
### Loading State Design
|
||||
|
||||
```tsx
|
||||
export function SearchLoadingModal({
|
||||
isOpen,
|
||||
timeout = 30,
|
||||
onTimeout,
|
||||
}: Props) {
|
||||
const [secondsElapsed, setSecondsElapsed] = useState(0);
|
||||
|
||||
useEffect(() => {
|
||||
if (!isOpen) return;
|
||||
|
||||
const interval = setInterval(() => {
|
||||
setSecondsElapsed((prev) => {
|
||||
if (prev >= timeout) {
|
||||
onTimeout();
|
||||
return prev;
|
||||
}
|
||||
return prev + 1;
|
||||
});
|
||||
}, 1000);
|
||||
|
||||
return () => clearInterval(interval);
|
||||
}, [isOpen, timeout, onTimeout]);
|
||||
|
||||
return (
|
||||
<div className="fixed inset-0 bg-black/50 flex items-center justify-center">
|
||||
<div className="bg-white p-8 rounded-lg max-w-md text-center">
|
||||
<Spinner className="mx-auto mb-4" />
|
||||
<p className="text-lg font-normal mb-2">Searching for specifications...</p>
|
||||
<p className="text-sm text-slate-500">
|
||||
{secondsElapsed}s / {timeout}s
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
### Error Handling UI
|
||||
|
||||
```tsx
|
||||
function SearchErrorModal({
|
||||
error,
|
||||
onRetry,
|
||||
onSkip,
|
||||
}: Props) {
|
||||
return (
|
||||
<div className="fixed inset-0 bg-black/50 flex items-center justify-center">
|
||||
<div className="bg-white p-8 rounded-lg max-w-md">
|
||||
<AlertCircle className="text-rose-500 mb-4 mx-auto" />
|
||||
<p className="text-lg font-normal mb-4">Search failed</p>
|
||||
<p className="text-sm text-slate-600 mb-6">{error}</p>
|
||||
<div className="flex gap-3">
|
||||
<button onClick={onRetry} className="flex-1 bg-primary text-white px-4 py-2 rounded">
|
||||
Retry Search
|
||||
</button>
|
||||
<button onClick={onSkip} className="flex-1 border border-slate-300 px-4 py-2 rounded">
|
||||
Skip
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Performance & Scalability Analysis
|
||||
|
||||
### Expected Latency Profile
|
||||
|
||||
| Component | Duration | Notes |
|
||||
|-----------|----------|-------|
|
||||
| AI extraction (Gemini/Claude) | 2-5s | Existing, cached |
|
||||
| Network request + HTML fetch | 2-8s | Highest variability |
|
||||
| HTML parsing (BeautifulSoup) | 100-500ms | |
|
||||
| Regex spec extraction | 10-50ms | |
|
||||
| **Total end-to-end** | **3-15s** | **Typical 20s with retries** |
|
||||
|
||||
### Handling Multiple Concurrent Searches
|
||||
|
||||
**Recommendation: Sequential Processing**
|
||||
- Process searches 1 at a time with 5-second delays between.
|
||||
- Prevents IP blocking and maintains consistent latency.
|
||||
- Max concurrent searches: 2-3 across all users.
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
# Global search queue
|
||||
search_queue: asyncio.Queue = asyncio.Queue()
|
||||
|
||||
async def process_search_queue():
|
||||
"""Background task: process queued searches sequentially."""
|
||||
while True:
|
||||
search_task = await search_queue.get()
|
||||
try:
|
||||
await search_and_extract(**search_task)
|
||||
finally:
|
||||
await asyncio.sleep(5) # Rate limit between searches
|
||||
search_queue.task_done()
|
||||
```
|
||||
|
||||
### Offline Graceful Degradation
|
||||
|
||||
**If no internet / search fails:**
|
||||
1. Catch all network exceptions.
|
||||
2. Return AI-extracted data only.
|
||||
3. Show UI message: `"Offline mode: using AI extraction only"`
|
||||
4. User proceeds with AI data (no pre-population from web search).
|
||||
5. **Optional:** Queue search for retry when connection restored.
|
||||
|
||||
### Rate Limiting to Avoid IP Blocks
|
||||
|
||||
**Per-IP Limits:**
|
||||
- Max 20 requests/minute to Google (distributed across all users).
|
||||
- Max 10 searches per user per minute.
|
||||
|
||||
**Backoff Strategy:**
|
||||
- First failure: Wait 2 seconds, retry once.
|
||||
- Second failure: Wait 10 seconds, mark IP as rate-limited.
|
||||
- If rate-limited: Return AI data, skip search for next 5 minutes.
|
||||
|
||||
**User-Agent Rotation:**
|
||||
- Rotate User-Agent on every request (10+ pool).
|
||||
- Prevents obvious bot detection.
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
**Cache by (part_number, category) for 24 hours:**
|
||||
```python
|
||||
@cache.cached(timeout=86400, key_prefix="spare_parts_search:")
|
||||
async def search_and_extract(part_number: str, category: str) -> SparePartSearchResult:
|
||||
# Expensive search operation
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Repeated searches for same part (e.g., "16GB RAM DDR4") hit cache.
|
||||
- Reduces network load and IP block risk.
|
||||
- Improves UX (instant pre-population on cached searches).
|
||||
|
||||
### Scalability Ceiling
|
||||
|
||||
**Current Estimate:**
|
||||
- Suitable for 50-100 item onboardings per day (10-20 searches/day).
|
||||
- Bottleneck: Google's IP blocking at ~20 requests/minute sustained.
|
||||
|
||||
**To Scale Beyond 100+ Searches/Day:**
|
||||
- Switch to **SerpAPI** ($50-200/month for high volume).
|
||||
- Implement **proxy rotation** (cost-effective, ~$5-20/month).
|
||||
- Use **manufacturer APIs directly** (Crucial, Kingston, Corsair offer product APIs).
|
||||
|
||||
---
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (Next.js) │
|
||||
│ AIOnboarding Component │
|
||||
│ │
|
||||
│ [Image Upload] → [AI Extraction] → [Confirm Item] │
|
||||
│ │ │
|
||||
│ v │
|
||||
│ [Show Search Loading Modal] │
|
||||
│ "Searching for specs..." (30s max) │
|
||||
│ │ │
|
||||
│ (on complete/error/timeout) │
|
||||
│ │ │
|
||||
│ [Pre-populate Fields] ← [Search Results] │
|
||||
│ Category / Type / Notes editable │
|
||||
│ │ │
|
||||
│ v │
|
||||
│ [User Reviews & Confirms] │
|
||||
│ │ │
|
||||
│ v │
|
||||
│ POST /api/onboarding/save │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ HTTP Request
|
||||
v
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Backend (FastAPI) │
|
||||
│ │
|
||||
│ POST /api/onboarding/extract │
|
||||
│ ├─ AI Extract (Gemini/Claude) │
|
||||
│ ├─ Check: category in whitelist + part_number? │
|
||||
│ └─ If YES: Call spare_parts_search.search_and_extract() │
|
||||
│ │ │
|
||||
│ v │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ SparePartsSearch Service │ │
|
||||
│ │ │ │
|
||||
│ │ Rate Limiter (token bucket, 0.2 req/sec) │ │
|
||||
│ │ │ │ │
|
||||
│ │ v │ │
|
||||
│ │ WebScraper (requests + User-Agent rotation) │ │
|
||||
│ │ ├─ search_google(query, timeout=10s) │ │
|
||||
│ │ └─ search_bing(query) [fallback] │ │
|
||||
│ │ │ │ │
|
||||
│ │ v │ │
|
||||
│ │ SpecExtractor (BeautifulSoup + regex) │ │
|
||||
│ │ ├─ Parse HTML → CSS selectors │ │
|
||||
│ │ ├─ Extract snippets │ │
|
||||
│ │ └─ Regex extraction: specs, manufacturer, etc. │ │
|
||||
│ │ │ │
|
||||
│ │ Cache (24h): (part_number, category) → specs │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ v │
|
||||
│ POST /api/onboarding/save │
|
||||
│ ├─ Save AI data + search results to Item │
|
||||
│ └─ Log to AuditLog │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation Strategies
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|-----------|
|
||||
| **Google IP blocking** | Search fails, no specs | Use Bing fallback, implement proxy rotation, cache results |
|
||||
| **Network timeout** | Slow UX, user frustration | 30s max timeout, show progress, fallback to AI data |
|
||||
| **Parsing failures** (HTML changes) | No spec extraction | Update regex patterns, use manufacturer APIs, human review |
|
||||
| **Rate limiting abuse** | Service degradation | Token bucket, per-user limits, exponential backoff |
|
||||
| **Search quality issues** | Wrong specs populated | Confidence scoring, human review before save, field editability |
|
||||
| **Offline (no internet)** | Feature unavailable | Graceful degradation, return AI data only, queue for retry |
|
||||
|
||||
---
|
||||
|
||||
## Testing & Validation Strategy
|
||||
|
||||
### Unit Tests
|
||||
|
||||
**File: `backend/tests/test_spare_parts_search.py`**
|
||||
|
||||
```python
|
||||
def test_search_and_extract_success():
|
||||
"""Test successful search and spec extraction."""
|
||||
result = await search_and_extract(
|
||||
part_number="Kingston Fury 16GB",
|
||||
category="RAM"
|
||||
)
|
||||
assert result.status == "success"
|
||||
assert result.specs["manufacturer"] == "Kingston"
|
||||
assert result.specs["capacity"] == "16GB"
|
||||
|
||||
def test_search_timeout():
|
||||
"""Test graceful timeout handling."""
|
||||
result = await search_and_extract(
|
||||
part_number="test",
|
||||
category="RAM",
|
||||
timeout=0.1 # Force timeout
|
||||
)
|
||||
assert result.status == "timeout"
|
||||
assert result.specs is None
|
||||
|
||||
def test_whitelist_matching():
|
||||
"""Test spare-part classification."""
|
||||
assert classify_as_spare_part("DDR4 RAM") == True
|
||||
assert classify_as_spare_part("CPU 16GB") == True
|
||||
assert classify_as_spare_part("Power Cable 6ft") == False
|
||||
assert classify_as_spare_part("Thermal Paste") == False
|
||||
|
||||
def test_spec_extraction_regex():
|
||||
"""Test regex patterns for spec extraction."""
|
||||
snippet = "Kingston Fury 16GB DDR4-3200 CAS 16"
|
||||
specs = extract_specs_from_snippet(snippet, category="RAM")
|
||||
assert specs["capacity"] == "16GB"
|
||||
assert specs["memory_type"] == "DDR4"
|
||||
assert specs["speed"] == "3200"
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
**File: `backend/tests/test_onboarding_with_search.py`**
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_onboarding_extract_with_search():
|
||||
"""Test full onboarding flow with search integration."""
|
||||
# Upload image
|
||||
response = await client.post(
|
||||
"/api/onboarding/extract",
|
||||
files={"file": ("test_ram.jpg", image_bytes)},
|
||||
data={"mode": "catalog"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["ai_data"]["category"] in ["RAM", "Memory"]
|
||||
assert data["search_status"] in ["success", "timeout", "error", "skipped"]
|
||||
|
||||
if data["search_status"] == "success":
|
||||
assert "manufacturer" in data["search_results"]
|
||||
assert "specs" in data["search_results"]
|
||||
```
|
||||
|
||||
### Frontend Tests
|
||||
|
||||
**File: `frontend/components/__tests__/SearchLoadingModal.test.tsx`**
|
||||
|
||||
```typescript
|
||||
describe("SearchLoadingModal", () => {
|
||||
it("displays countdown timer", () => {
|
||||
render(<SearchLoadingModal isOpen timeout={30} />);
|
||||
expect(screen.getByText(/Searching for specifications/)).toBeInTheDocument();
|
||||
expect(screen.getByText(/0s \/ 30s/)).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("calls onTimeout after timeout expires", async () => {
|
||||
const onTimeout = vi.fn();
|
||||
render(<SearchLoadingModal isOpen timeout={1} onTimeout={onTimeout} />);
|
||||
|
||||
await new Promise(resolve => setTimeout(resolve, 1100));
|
||||
expect(onTimeout).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Field Testing with Users
|
||||
|
||||
1. **Recruit 3-5 power users** (heavy inventory users).
|
||||
2. **Phase A (1 week)**: Manual specification lookup (baseline).
|
||||
3. **Phase B (1 week)**: Test Phase 4.1 with automatic search.
|
||||
4. **Metrics**:
|
||||
- Time-to-save per item (before vs. after).
|
||||
- Accuracy of auto-populated fields.
|
||||
- Number of user edits post-search.
|
||||
- Search success rate (not timeout/error).
|
||||
5. **Collect feedback**: Desired fallback sources, UX tweaks, edge cases.
|
||||
|
||||
---
|
||||
|
||||
## RESEARCH COMPLETE
|
||||
|
||||
Reference in New Issue
Block a user