I'm running a crawler that does a lot of recursion, so much that at some point, the stored session information in Redis expires, and the scraper stops using the configured proxy.
This could be solved by increasing the expiration time:
|
conn.set(key, session, ex=REDIS_SHORT) |
I'm running a crawler that does a lot of recursion, so much that at some point, the stored session information in Redis expires, and the scraper stops using the configured proxy.
This could be solved by increasing the expiration time:
memorious/memorious/logic/http.py
Line 98 in da7291a