docs(4.1): capture phase context and discussion log for AI spare parts deep identification

- AI Prompt: Detailed categorization to distinguish spare parts from consumables - Search: Web scraping (requests + BeautifulSoup) to extract Google results, no API key - Trigger: Automatic background search after AI extraction if category matches whitelist - UX: Block until search completes, user reviews and edits all fields before save - Mapping: Extract product type/specs/manufacturer/description to Notes field - Retry on failure, user can skip if needed
2026-04-22 16:16:51 +03:00
parent d3f8b6c4ff
commit b90085cce5
2 changed files with 299 additions and 0 deletions
--- a/.planning/phases/4.1-ai-spare-parts-deep-id/4.1-CONTEXT.md
+++ b/.planning/phases/4.1-ai-spare-parts-deep-id/4.1-CONTEXT.md
@@ -0,0 +1,129 @@
+# Phase 4.1: AI Prompt Enhancement — Spare Parts Deep Identification - Context
+
+**Gathered:** 2026-04-22
+**Status:** Ready for planning
+
+<domain>
+## Phase Boundary
+
+Enhance AI extraction pipeline to automatically identify spare parts (vs consumables) and search the internet for detailed specifications. When a Part Number is detected on an identified spare part, extract product type, specifications, manufacturer, and description from search results, pre-populate Item fields for user review, and allow editing before save.
+
+**In scope:**
+- Update AI prompt to distinguish spare parts (RAM, SSD, NVME, PCIe, disk, etc.) from consumables (cords, connectors, small hardware)
+- Implement web-based internet search for part specifications (no API key required)
+- Extract specs, manufacturer, product type, and descriptions from search results
+- Pre-populate Item Category/Type/Notes fields with search results for user review
+- Implement retry logic and error handling for search failures
+- Validate with field users from Phase 4 deployments
+
+**Out of scope:**
+- Caching of search results (can be deferred)
+- Price estimation from search (optional enhancement)
+- Multi-language support for search results
+- Local database of part specifications (MVP uses web search only)
+
+</domain>
+
+<decisions>
+## Implementation Decisions
+
+### AI Prompt Enhancement
+- **D-01:** No explicit spare-part classification field. Infer from extracted category using a backend whitelist of known spare-part categories.
+- **D-02:** Use detailed categorization logic in the AI prompt: "If a component plugs into or connects to another device (not just cabling), classify as spare part." Provide comprehensive examples (RAM, SSD, NVME, PCIe cards, disks, memory modules, processors, etc.) vs consumables (cords, connectors, adhesives, small fasteners).
+
+### Internet Search Integration
+- **D-03:** Use web scraping with Python `requests` + `BeautifulSoup` to extract Google search results. No API key required, suitable for low volume (tens of items per hour maximum).
+- **D-04:** Implement rate limiting with delays and User-Agent headers to avoid IP blocking by Google. Details to be determined during planning (suggested: 1-2 second delay between requests, rotating User-Agent).
+
+### Search Trigger & User Flow
+- **D-05:** Automatic background search: After AI extraction, if extracted category matches the spare-parts whitelist AND Part Number is present, trigger internet search automatically.
+- **D-06:** Block onboarding UI until search completes. Show loading state during search.
+- **D-07:** On search failure: Display error message with "[Retry]" and "[Skip]" buttons. User can retry or proceed without specs.
+- **D-08:** User reviews all search-populated fields before final save. User can edit any incorrect/incomplete data in the form.
+
+### Data Extraction & Item Mapping
+- **D-09:** Extract from search results: product type/category, specifications (capacity, speed, voltage, etc.), manufacturer/model name, and detailed description.
+- **D-10:** Store extracted data:
+  - **Category/Item Type:** Pre-populate with refined values from search. User can edit before saving.
+  - **Notes field:** Store detailed specs, manufacturer, description, and any other details from search.
+- **D-11:** Item Type field remains searchable/concise (e.g., "RAM DDR4" not full spec). Detailed specs go in Notes.
+
+### Claude's Discretion
+- Specific spare-parts category whitelist (which categories trigger search — to be built from field feedback and product categorization)
+- Search timeout duration (recommended: 15-30 seconds max before showing "no results" error)
+- BeautifulSoup parsing logic and CSS selectors for Google search results (site-specific and may need tuning)
+- Retry logic details (number of retries, backoff strategy)
+- Fallback behavior if internet is unavailable (graceful degradation — show empty spec fields for user to fill)
+
+</decisions>
+
+<canonical_refs>
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Core Architecture & Data Models
+- `PROJECT_ARCHITECTURE.md` — Item model fields (Category, Type, Notes), AI integration (Gemini 2.0 Flash, Claude 3.5 Sonnet), multi-AI provider pattern
+- `PROJECT.md` — Multi-AI provider flexibility requirement, offline-first constraint, UI fidelity standards (no UPPERCASE, no BOLD fonts)
+- `.planning/REQUIREMENTS.md` — Mobile UX, field user validation requirements
+
+### AI & Prompt Design
+- `backend/ai/gemini_extractor.py` — Current Gemini prompt structure and extraction pattern
+- `backend/ai/claude_extractor.py` — Current Claude fallback pattern and prompt structure
+
+### Frontend Integration (Onboarding Flow)
+- `frontend/components/AIOnboarding.tsx` — Current item extraction and confirmation flow; where search results will be integrated
+
+### UI/UX Standards
+- `dev_docs/` — Premium fidelity standards (Tailwind, Lucide, no UPPERCASE, no BOLD)
+
+No external specification documents — requirements fully captured in decisions above.
+
+</canonical_refs>
+
+<code_context>
+## Existing Code Insights
+
+### Reusable Assets
+- **AI Extractor Pattern:** `backend/ai/gemini_extractor.py` and `claude_extractor.py` provide the extraction interface. Search integration can follow the same async pattern.
+- **AIOnboarding Component:** `frontend/components/AIOnboarding.tsx` already manages item confirmation flow. Search results will integrate into the review-and-edit phase before save.
+- **ConfigManager:** `backend/config_manager.py` handles runtime configuration. Can be extended for search preferences (rate limits, timeout).
+- **Admin Dashboard:** `frontend/components/AdminDashboard.tsx` has patterns for secure field masking; future enhancement for Google Search settings.
+
+### Established Patterns
+- **Multi-AI Provider:** Backend already switches between Gemini and Claude. Search integration is independent but should use the same provider-agnostic pattern if extending AI for parsing search results.
+- **Offline-First:** Sync uses UUID idempotency. Search is online-only; gracefully skip if network unavailable.
+- **Error Handling:** Admin dashboard shows error states. Onboarding should follow similar patterns for search failures.
+
+### Integration Points
+- **AI Extraction:** Search triggers after AI extraction completes (in `AIOnboarding.tsx`)
+- **Item Save:** Search results pre-populate Item fields; user edits then saves as normal
+- **Backend:** `/items/` POST endpoint receives search-enriched Item data
+
+</code_context>
+
+<specifics>
+## Specific Ideas
+
+- **Field User Validation (Phase 4):** Deploy with field teams running Phase 4 to gather feedback on search accuracy and relevance. Use their corrections to refine the spare-parts whitelist and prompt.
+- **Spare-Parts Whitelist:** Build from common warehouse components: RAM, SSD, NVME, PCIe cards, CPU, power supplies, network cards, storage controllers, motherboards. Will refine based on field feedback.
+- **Web Scraping Resilience:** Include user-agent rotation and request delays to avoid Google blocks. Consider fallback to a second search engine (e.g., Bing) if Google scraping fails.
+
+</specifics>
+
+<deferred>
+## Deferred Ideas
+
+- **Price Estimation:** Extract approximate cost from search results for asset valuation. Deferred to Phase 5 (nice-to-have, adds complexity).
+- **Search Result Caching:** Cache search results for repeated part numbers to reduce API calls. Deferred to Phase 5 (optimization, not MVP).
+- **Multi-Language Search:** Support searching in multiple languages based on user locale. Deferred to Phase 6+ (localization out of scope for v2).
+- **Local Part Database:** Build local cache of known parts to avoid repeated searches. Deferred to Phase 6+ (requires significant infrastructure).
+
+None — discussion stayed within phase scope.
+
+</deferred>
+
+---
+
+*Phase: 4.1-ai-spare-parts-deep-id*
+*Context gathered: 2026-04-22*
--- a/.planning/phases/4.1-ai-spare-parts-deep-id/4.1-DISCUSSION-LOG.md
+++ b/.planning/phases/4.1-ai-spare-parts-deep-id/4.1-DISCUSSION-LOG.md
@@ -0,0 +1,170 @@
+# Phase 4.1: AI Prompt Enhancement — Spare Parts Deep Identification - Discussion Log
+
+> **Audit trail only.** Do not use as input to planning, research, or execution agents.
+> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
+
+**Date:** 2026-04-22
+**Phase:** 4.1-ai-spare-parts-deep-id
+**Areas discussed:** AI Prompt Strategy, Search API Selection, Search Trigger & Confirmation, Data Extraction & Item Mapping
+
+---
+
+## AI Prompt Strategy
+
+### Classification Approach
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Yes, explicit classification | AI returns spare_part_detected field. Backend auto-searches if true. Cleaner, deterministic behavior. | |
+| No, infer from category | Use the extracted category to infer if it's likely a spare part (e.g., 'RAM', 'SSD' → search). Less explicit, fewer prompt changes. | ✓ |
+| Hybrid approach | AI provides a classification + confidence score. Backend uses both to decide whether to search. | |
+
+**User's choice:** No, infer from category
+
+**Rationale:** Keeps the AI extraction unchanged; the backend maintains a whitelist of spare-part categories that trigger search. Simpler to implement and maintain.
+
+---
+
+### Prompt Detail Level
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Basic list in prompt | Add simple guidance: 'Spare parts: RAM, SSD, NVME, PCIe cards, etc. Consumables: cords, connectors, small hardware. Extract category accordingly.' | |
+| Detailed categorization | Provide extensive examples and decision logic: 'If it's a component that plugs into or connects to another device (not just cabling), classify as spare part.' | ✓ |
+| You decide | Claude picks the right level of detail based on testing during planning phase. | |
+
+**User's choice:** Detailed categorization
+
+**Rationale:** Field users need reliable distinction. Detailed prompt with decision logic (plugs/connects vs just cables) reduces false positives on consumables.
+
+---
+
+## Search API Selection
+
+### Internet Search Service
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Google Custom Search (official API) | Reliable, well-documented, ~100 free queries/day + paid tiers. Requires API key + setup. | |
+| Bing Search API | Fast, cheaper than Google ($0-7/month for 1M queries), fewer free tier limits. Less common in Python examples. | |
+| DuckDuckGo or free Python libs | Zero cost, no auth, easier to start. Less reliable (may get blocked, unstable parsing). Good for MVP, risky for production. | |
+| Deferred decision | Research both during planning phase, decide based on cost/complexity/reliability trade-offs. | |
+
+**User's choice:** *Modified to avoid API keys*
+
+**User's note:** "Google Custom Search needs an api key? If yes, I don't need this. We cannot use free Google search? The items will not be 1000s in short time, will be tens in an hour maximum rate."
+
+**Follow-up decision:** User prioritizes zero API key overhead. Volume is low (tens/hour). Recommend web scraping instead of official API.
+
+### Search Implementation Method
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Web scraping (requests + BeautifulSoup) | No API key needed, zero cost, works for low volume (tens/hour). Risk: Google may block scraper IPs. Mitigate with rate limiting + delays. | ✓ |
+| DuckDuckGo instant answer API | Free, no key, returns structured 'instant answers' for simple queries (specs, product info). Limited but sufficient for parts lookup. | |
+| Python library (googlesearch-py) | Open-source, zero cost, handles Google scraping internally. Less reliable than official API but good for MVP. | |
+
+**User's choice:** Yes, web scraping
+
+**Rationale:** Zero API key overhead, suitable for low-volume field use, can implement rate limiting to avoid blocking.
+
+---
+
+## Search Trigger & Confirmation
+
+### When Search Should Occur
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Automatic in background | After AI extraction, if category is in spare-part whitelist and Part Number exists, search automatically. Results appear in UI shortly after. | ✓ |
+| User-initiated button | Show 'Search for specs' button in the onboarding UI. User clicks to trigger search. More control, less friction-free. | |
+| Auto-search + optional repeat | Search auto-triggers by default. User can click 'Refresh search' to get fresh results if needed. | |
+
+**User's choice:** Automatic in background
+
+**Rationale:** Frictionless for field users. Reduces decision fatigue; specs appear automatically if available.
+
+---
+
+### UI Behavior During Search
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Non-blocking (populate later) | Show item form immediately. Specs from search fill in after they arrive. User can save without waiting for search. | |
+| Optional block (wait or skip) | Show loading state. Button to 'Save anyway' or 'Wait for specs'. User chooses based on impatience. | |
+| Quick timeout (3-5 sec) | Wait max 3-5 seconds for search results. If no results arrive, continue without them. Prevents user frustration from slow internet. | |
+
+**User's choice (modified):** "User will wait for all fields to be populated, and if not ok, will edit not ok fields and after that will save the new item in inventory."
+
+**Rationale:** Review-and-edit-before-save model. User blocks until search completes, reviews all pre-populated fields, edits as needed, then saves.
+
+**Implication:** Requires a reasonable timeout before showing "no results" error; details to be determined during planning.
+
+---
+
+### Failure Handling
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Show error, let user retry | Display 'Search failed. [Retry] or [Skip]'. User can retry or proceed without specs. | ✓ |
+| Pre-fill with manual entry | Search fails → show empty spec fields. User manually enters details they know. No retry. | |
+| Reasonable timeout (15 sec) then skip | Wait 15 seconds max. If no results, show 'No specs found online. [Edit manually]' and continue. | |
+
+**User's choice:** Show error, let user retry
+
+**Rationale:** User has control. Can retry if network is temporarily unavailable; can skip if they don't want to wait.
+
+---
+
+## Data Extraction & Item Mapping
+
+### Fields to Extract from Search Results
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Product type/category | What the part is (RAM, SSD, etc.). Refines the AI-extracted category if needed. | ✓ |
+| Specifications (speed, capacity, voltage) | Technical details that matter for inventory (DDR4 32GB, 3.0TB SSD, etc.). | ✓ |
+| Manufacturer/model | Brand and model name if found. Helps distinguish between variants. | ✓ |
+| Price estimate | Approx cost if available. Useful for valuation, but may be outdated or region-specific. | |
+
+**User's choice:** Product type, Specifications, Manufacturer/model, plus "details/description of that item too"
+
+**Rationale:** Comprehensive data about each part. Price optional; description/details more useful than price for inventory accuracy.
+
+---
+
+### Item Field Mapping
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Enrich 'Item Type' field | Item Type becomes detailed: 'RAM DDR4 32GB 3000MHz' (combining specs + type). Category stays as selected. | |
+| Use 'Notes' for detailed specs | Item Type is simpler (e.g., 'RAM'). Notes field gets the detailed specs and description from search. | ✓ |
+| Both fields | Item Type is searchable summary ('RAM DDR4'). Notes gets full detailed specs/description/manufacturer. | |
+
+**User's choice:** Use 'Notes' for detailed specs
+
+**Rationale:** Keeps Item Type concise and searchable. Notes field captures all detailed information without cluttering the type field.
+
+---
+
+### Category Refinement from Search
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Pre-populate, user can edit | Search results suggest a refined category/type. User can accept or change it before saving. | ✓ |
+| Trust AI extraction | Keep the AI's original category/type. Search results fill in Notes only. No second-guessing the AI. | |
+| Suggest if high confidence | If search results clearly indicate a different category (e.g., search says 'SSD' but AI said 'Storage'), suggest it. Otherwise keep AI extraction. | |
+
+**User's choice:** Pre-populate, user can edit
+
+**Rationale:** Search often clarifies or refines the category. User can accept the refined value or revert to AI extraction if search is incorrect.
+
+---
+
+## Claude's Discretion
+
+Areas where user deferred to Claude for implementation decisions:
+- Specific spare-parts category whitelist (to be built from field feedback)
+- Search timeout duration (suggested: 15-30 seconds before showing error)
+- BeautifulSoup parsing logic and CSS selectors for Google results
+- Rate limiting strategy (delays, retries, backoff)
+- Fallback behavior if internet is unavailable
+
+---
+
+## Deferred Ideas
+
+- **Price Estimation** — Extract approximate cost from search results for asset valuation. Noted for Phase 5 (nice-to-have).
+- **Search Result Caching** — Cache results for repeated part numbers to reduce searches. Noted for Phase 5 (optimization, not MVP).
+- **Multi-Language Search** — Support multiple languages. Noted for Phase 6+ (localization).
+- **Local Part Database** — Build local cache of known parts. Noted for Phase 6+ (infrastructure heavy).