# Phase 4.1 Research: AI Prompt Enhancement — Spare Parts Deep Identification **Research Date:** 2026-04-22 **Scope:** Web scraping implementation, spare-parts classification, AI prompt enhancement, search result parsing, backend/frontend integration, and performance/scalability. --- ## 1. Web Scraping Best Practices: Python Requests + BeautifulSoup ### Key Findings **Approach & Risks:** - **Direct Google scraping** is technically feasible but risky: Google actively detects and blocks scrapers with 429 (Too Many Requests) errors, CAPTCHA challenges, and IP bans. - **Terms of Service violation**: Google's ToS explicitly forbids scraping search results. - **HTML structure volatility**: Google changes CSS selectors and HTML markup frequently, breaking scrapers. - **Practical reality**: Direct scraping works for low-volume scenarios (tens of requests/hour) with proper mitigations. **Safer Alternatives:** 1. **SerpAPI / Similar APIs**: Officially maintained, handles blocking/rotation, but costs money ($5-50/month depending on volume). 2. **Bing scraping**: Less aggressively blocked than Google, similar HTML structure, viable fallback. 3. **Manufacturer sites** (Dell, HP, Kingston, Crucial): Most reliable source for spare-part specs. 4. **GitHub Issues / StackOverflow**: Often contain real-world component usage and specifications. **Recommended Hybrid Approach:** - Primary: Search manufacturer specs directly (most accurate). - Fallback 1: Bing web search with BeautifulSoup. - Fallback 2: Google search (if Bing returns no results). - Fallback 3: Return AI-extracted data only (graceful offline degradation). ### Rate Limiting Strategies **Implementation:** - **Delay between requests**: 2-5 seconds minimum (random jitter recommended). - **User-Agent rotation**: Cycle through 10+ realistic User-Agent strings (Chrome, Firefox, Safari across Windows/Mac/Linux). - **Exponential backoff**: 1s → 2s → 4s → 8s → fail. - **Token bucket algorithm**: Max 0.2 requests/second (1 request per 5 seconds) per IP. **User-Agent Pool (Examples):** ``` Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (Chrome/120.0.0.0) Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (Firefox/121.0) Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (Safari/537.36) ``` ### Error Handling & Timeout Strategies **HTTP Status Codes:** - **429 (Too Many Requests)**: Wait 10 seconds, retry once, then fail gracefully. - **403 (Forbidden)**: IP blocked; rotate User-Agent, increase delay, or skip. - **500+ (Server Error)**: Retry with exponential backoff. - **Timeout (>10s)**: Abort search, return AI data only, log warning. **CAPTCHA Detection:** - BeautifulSoup can detect CAPTCHA forms by checking for `