Add AutoThrottle and retry settings to enhance crawling efficiency and politeness

Author: DedSec2050

.env.example

@@ -7,6 +7,15 @@ LOCAL_DEDUP_FALLBACK_ENABLED=true
 LOCAL_DEDUP_STATE_PATH=./out/ingestion.seen.json
 INCREMENTAL_ENABLED=true
 STATE_STORE_PATH=./out/source_state.sqlite
+AUTOTHROTTLE_ENABLED=true
+AUTOTHROTTLE_START_DELAY=1
+AUTOTHROTTLE_MAX_DELAY=10
+AUTOTHROTTLE_TARGET_CONCURRENCY=1
+DOWNLOAD_DELAY=1
+CONCURRENT_REQUESTS=8
+CONCURRENT_REQUESTS_PER_DOMAIN=2
+RETRY_ENABLED=true
+RETRY_TIMES=3
 METRICS_ENABLED=true
 METRICS_OUTPUT_PATH=./out/metrics/ingestion_metrics.jsonl
 VALIDATION_ENABLED=true

README.md

@@ -13,6 +13,7 @@ What this project currently does:
 - Normalizes output records into a common ingestion envelope.
 - Publishes events to Kafka (production path) or JSONL (local development path).
 - Applies local deduplication using stable IDs with fallback fingerprint dedup when IDs are missing.
+- Uses conservative AutoThrottle and retry defaults so crawls stay polite on shared/public sources.
 - Persists source crawl state in SQLite so repeated runs are efficient and idempotent.
 
 What this repository does not include yet:
@@ -57,6 +58,7 @@ Completed:
 - [x] Structured item-ingestion logs (JSON log lines)
 - [x] Backfill execution mode in startup script (`--backfill`)
 - [x] Pydantic schema validation in ingestion pipeline (configurable mode)
+- [x] AutoThrottle and retry defaults for polite crawling
 - [x] Automated startup script with `--all`, `--spider`, `--reset-state`, `--skip-install`
 - [x] Docker and GitHub Actions scaffolding
 
@@ -175,6 +177,15 @@ The script uses the local JSONL sink by default. To route output to Kafka, set `
 - LOCAL_DEDUP_STATE_PATH: sidecar file that stores seen dedup keys
 - INCREMENTAL_ENABLED: turn source-level incremental crawling on/off
 - STATE_STORE_PATH: SQLite state file used for source watermarks and HTTP cache headers
+- AUTOTHROTTLE_ENABLED: enable Scrapy AutoThrottle (default: true)
+- AUTOTHROTTLE_START_DELAY: initial delay in seconds for AutoThrottle (default: 1)
+- AUTOTHROTTLE_MAX_DELAY: max delay in seconds for AutoThrottle (default: 10)
+- AUTOTHROTTLE_TARGET_CONCURRENCY: target concurrent requests per server (default: 1)
+- DOWNLOAD_DELAY: fixed download delay used alongside AutoThrottle (default: 1)
+- CONCURRENT_REQUESTS: total concurrent requests across the crawler (default: 8)
+- CONCURRENT_REQUESTS_PER_DOMAIN: concurrent requests per domain (default: 2)
+- RETRY_ENABLED: enable retries for transient HTTP failures (default: true)
+- RETRY_TIMES: number of retry attempts for transient HTTP failures (default: 3)
 - METRICS_ENABLED: emit per-spider run metrics to JSONL file (default: true)
 - METRICS_OUTPUT_PATH: JSONL path for ingestion run metrics
 - VALIDATION_ENABLED: enable or disable Pydantic validation in the pipeline (default: true)

src/bioscope_ingestion/settings.py

@@ -1,10 +1,23 @@
+from common.config import env_bool, env_int
+
 BOT_NAME = "bioscope_ingestion"
 
 SPIDER_MODULES = ["bioscope_ingestion.spiders"]
 NEWSPIDER_MODULE = "bioscope_ingestion.spiders"
 
 ROBOTSTXT_OBEY = True
 
+AUTOTHROTTLE_ENABLED = env_bool("AUTOTHROTTLE_ENABLED", True)
+AUTOTHROTTLE_START_DELAY = env_int("AUTOTHROTTLE_START_DELAY", 1)
+AUTOTHROTTLE_MAX_DELAY = env_int("AUTOTHROTTLE_MAX_DELAY", 10)
+AUTOTHROTTLE_TARGET_CONCURRENCY = float(env_int("AUTOTHROTTLE_TARGET_CONCURRENCY", 1))
+DOWNLOAD_DELAY = float(env_int("DOWNLOAD_DELAY", 1))
+CONCURRENT_REQUESTS = env_int("CONCURRENT_REQUESTS", 8)
+CONCURRENT_REQUESTS_PER_DOMAIN = env_int("CONCURRENT_REQUESTS_PER_DOMAIN", 2)
+RETRY_ENABLED = env_bool("RETRY_ENABLED", True)
+RETRY_TIMES = env_int("RETRY_TIMES", 3)
+RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]
+
 ITEM_PIPELINES = {
     "bioscope_ingestion.pipelines.KafkaPipeline": 300,
 }