What this prompt does
This prompt asks the AI to build a production-ready Python web scraper rather than a throwaway script. It specifies seven concrete requirements: parsing with [library], retry logic with exponential backoff capped at [max_retries], rate limiting at [requests_per_second], handling of 403/429/timeout/CAPTCHA errors, structured output as [output_format], rotating-file logging, and pagination up to [max_pages] — all with type hints and docstrings.
The structure works because it front-loads the unglamorous reliability concerns that decide whether a scraper survives unattended. The [target_site_type] and [data_fields] variables tell the AI what to extract and from what shape of page, while [max_retries], [requests_per_second], and [max_pages] give it the operational guardrails — so you get politeness and resilience baked in, not bolted on after the first 429 ban.
When to use it
- You need a scraper that runs unattended and recovers from transient errors instead of crashing.
- You're extracting structured
[data_fields]from a paginated site and want clean CSV/JSON out. - You want rate limiting and backoff in place from the start to stay polite and avoid bans.
- You need rotating logs so a long-running scrape is debuggable after the fact.
- You're choosing between BeautifulSoup and Scrapy and want the requirements to drive the choice.
Example output
Expect a complete, runnable Python module: a scraper class or functions using [library], a retry decorator or loop with exponential backoff, a rate limiter, error handlers branching on status codes, and an output writer for [output_format]. Logging is wired through a rotating file handler, pagination loops to [max_pages], and every function carries type hints and docstrings — code you can drop in and configure, not pseudocode.
Pro tips
- Match
[library]to the site: static HTML suits BeautifulSoup + requests, while JS-heavy or large crawls justify Scrapy. - Keep
[requests_per_second]conservative (the default of 2 is polite) — aggressive rates are what trigger 429s and IP bans. - Be specific with
[data_fields]; vague fields produce brittle selectors the AI guesses at. - Set
[max_pages]to a real bound so a runaway loop can't hammer the target indefinitely. - Ask for both CSV and JSON in
[output_format]if downstream consumers differ — it's cheap to emit both. - The prompt mentions CAPTCHA handling but no scraper truly solves CAPTCHAs; treat that as graceful detection and backoff, not bypass.
