Scrapy middlewares that reads proxy config from settings.
pip install proxy-middleware
A middleware that sets proxy from settings file.
Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:
DOWNLOADER_MIDDLEWARES = {
'proxy_middleware.ProxyFromSettingsMiddleware': 10,
...
Pass proxy config via HTTP_PROXY and HTTPS_PROXY settings
variables. HTTPPROXY_AUTH_ENCODING is also respected:
scrapy crawl my-spider -s HTTP_PROXY=http://localhost:8118
A middleware that proxies only requests to TOR sites (domain ends with ".onion").
Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:
DOWNLOADER_MIDDLEWARES = {
'proxy_middleware.ProxyOnlyTorMiddleware': 10,
...
Settings: HTTP_PROXY for an http proxy and HTTPS_PROXY for an https proxy.
License is MIT.
