{"id":"web-scraper-api","name":"web-scraper-api","summary":"自動アンチボットバイパスを備えた本番レベルのウェブスクレイピング、40+ターゲット向けの構造化JSON解析、ジオターゲティング。","body":"# Oxylabs Web Scraper API\n\n## Authentication\n\nRequires HTTP Basic Auth with credentials from environment variables:\n\n```bash\ncurl -u \"$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD\" ...\n```\n\n## Endpoint\n\n```\nPOST https://realtime.oxylabs.io/v1/queries   # immediate response\nPOST https://data.oxylabs.io/v1/queries       # Push-Pull jobs, callbacks, storage\nContent-Type: application/json\n```\n\n## Core Parameters\n\n| Parameter | Required | Description |\n|-----------|----------|-------------|\n| `source` | Yes | Target scraper (e.g., `universal`, `amazon_product`, `google_search`) |\n| `url` | Conditional | URL to scrape (for `universal` and `*_url` sources) |\n| `query` | Conditional | Search query or product ID (for `*_search` and `*_product` sources) |\n| `parse` | No | Enable structured data parsing (recommended for supported sources) |\n| `render` | No | JavaScript rendering: `html` or `png` |\n| `geo_location` | No | Geographic targeting: country/state/city, ZIP/postcode, coordinates, or Criteria ID where supported |\n| `session_id` | No | Reuse the same proxy IP across multiple jobs |\n| `content_encoding` | No | Set to `base64` when downloading image files via Realtime or Push-Pull |\n| `user_agent_type` | No | Device/browser preset, e.g., `desktop_chrome`, `mobile_ios`, `tablet_android` |\n| `locale` | No | Interface language / `Accept-Language`, e.g., `de-DE` |\n| `callback_url` | No | Push-Pull callback endpoint |\n| `storage_type`, `storage_url` | No | Push-Pull cloud upload target (`gcs`, `s3`, `tos`, `s3_compatible`) |\n| `markdown`, `xhr` | No | Enable markdown or captured XHR result types |\n| `browser_instructions` | No | Rendered browser actions; requires `render: \"html\"` |\n| `parsing_instructions`, `parser_preset` | No | Custom parser rules or saved preset; pair with `parse: true` |\n| `client_notes` | No | Client-side job tag saved with the job metadata |\n| `domain`, `subdomain`, `start_page`, `pages`, `limit`, `store_id`, `delivery_zip`, `fulfillment_type` | Source-specific | Marketplace/search/store localization and pagination fields |\n\n`user_agent_type` values: `desktop`, `desktop_chrome`, `desktop_edge`, `desktop_firefox`, `desktop_opera`, `desktop_safari`, `mobile`, `mobile_android`, `mobile_ios`, `tablet`, `tablet_android`, `tablet_ios`.\n\n## Context Parameters\n\nAdd these as `{ \"key\": \"...\", \"value\": ... }` objects in `context`:\n\n| Key | Use |\n|-----|-----|\n| `force_headers`, `headers` | Merge custom headers with managed headers |\n| `force_cookies`, `cookies` | Merge custom cookies with managed cookies |\n| `http_method`, `content` | Use `post` with Base64-encoded body content |\n| `follow_redirects` | Follow 3xx redirect chains |\n| `successful_status_codes` | Treat specific non-standard HTTP codes as successful |\n\nFor multi-format output, enable types in the payload (`parse`, `markdown`, `xhr`, `render: \"png\"`) and request them with `?type=raw,parsed,png,markdown,xhr`.\n\nFor batch Push-Pull jobs, use `POST /v1/queries/batch` with arrays only for `query` or `url`; keep all other parameters singular. Maximum batch size is 5,000 values.\n\n## Quick Start\n\n**Scrape any URL:**\n```bash\ncurl -X POST 'https://realtime.oxylabs.io/v1/queries' \\\n  -u \"$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD\" \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"source\": \"universal\", \"url\": \"https://example.com\"}'\n```\n\n**Google search with parsing:**\n```bash\ncurl -X POST 'https://realtime.oxylabs.io/v1/queries' \\\n  -u \"$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD\" \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"source\": \"google_search\", \"query\": \"best laptops\", \"parse\": true}'\n```\n\n**Amazon product by ASIN:**\n```bash\ncurl -X POST 'https://realtime.oxylabs.io/v1/queries' \\\n  -u \"$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD\" \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"source\": \"amazon_product\", \"query\": \"B07FZ8S74R\", \"parse\": true}'\n```\n\n## Choosing the Right Source\n\n1. **Use specific sources when available** (`amazon_product`, `google_search`) - better parsing and reliability\n2. **Use `universal` for unsupported sites** - works with any URL\n3. **Enable `parse: true`** for structured JSON output on supported sources\n\n## Response Structure\n\n```json\n{\n  \"results\": [{\n    \"content\": \"...\",\n    \"status_code\": 200,\n    \"url\": \"https://...\"\n  }]\n}\n```\n\nWith `parse: true`, `content` contains structured data (title, price, reviews, etc.) instead of raw HTML.\n\n## Available Sources\n\nFor the complete list of 40+ supported sources organized by category, see [sources.md](sources.md).\n\n## More Examples\n\nFor detailed request/response examples including geo-location, JavaScript rendering, and custom headers, see [examples.md](examples.md).\n\n## Error Handling\n\n| Code | Meaning |\n|------|---------|\n| 200 | Success |\n| 400 | Invalid parameters |\n| 401 | Authentication failed |\n| 403 | Access denied |\n| 429 | Rate limit exceeded |\n\n## Key Guidelines\n\n- Always set `parse: true` for supported sources to get structured data\n- Use ZIP codes for US e-commerce geo-location (e.g., `\"90210\"`)\n- Use country/state format for search engines (e.g., `\"California,United States\"`)\n- Add `render: \"html\"` for JavaScript-heavy pages\n- Use `render: \"\"` only to disable automatic forced rendering for force-rendered pages; set client timeouts near 180 seconds for rendered Realtime or Proxy Endpoint requests\n- Add `content_encoding: \"base64\"` when scraping image URLs, then decode `results[0].content` before saving the file","author":"@oxylabs","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/oxylabs/agent-skills/tree/main/skills/web-scraper-api","license":"MIT","category":null,"lang":"en","tokens":1429,"stars":0,"calls30d":1,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"examples.md","size":6041,"sha256":"fabe314a1c55e472e99fc9fdb2237fc43c1c99f6a1e38a91b7c645d8bea10659"},{"path":"sources.md","size":5501,"sha256":"a5ce5712234e5512e65f26ec697f40c562754dff9c46f3bd15cd6d7e1c4d710e"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["data.oxylabs.io","realtime.oxylabs.io"]}}