Rate limiting / request throttling #2
Description
Feature Request
Add built-in rate limiting to prevent overwhelming target servers during batch extraction.
Options to consider:
- Per-domain request interval (e.g., 1 request/second per domain)
- Global concurrency limit
- Configurable via parameter or environment variable
Motivation
Currently documented as user responsibility in Disclaimer. Built-in support would make production use safer.