Compatible mode in 3.7.8 is not compatible with 3.1.0 release

[davdat]

Summary.

cloudscraper version: 3.1.0

scraper = cloudscraper.create_scraper(debug=False)
scraper.get("https://babel.hathitrust.org/cgi/pt?id=hvd.hn5gdg&seq=11")

Status: 200. Length: 24855. Time: 2.2622790336608887

cloudscraper version: 3.7.8

scraper = cloudscraper.create_compat_scraper()
scraper.get("https://babel.hathitrust.org/cgi/pt?id=hvd.hn5gdg&seq=11")

Status: 403. Length: 7432. Time: 0.42971038818359375

Expected Result

3.7.8 scraper should result in a 200 status/successful response

[zinzied]

"Fixed in v3.8.0. The issue was caused by the new TLS fingerprinting mechanism in 3.7.x triggering WAF blocks on sites like HathiTrust.

We have disabled the problematic tls-chameleon dependency by default in v3.8.0. The standard cloudscraper.create_scraper() now uses the robust requests-based method (same as v3.1.0) and bypasses HathiTrust successfully (Status 200).

Solution:

Upgrade to v3.8.0: pip install --upgrade ai-cloudscraper
Use the standard scraper:
python
scraper = cloudscraper.create_scraper()
scraper.get('https://babel.hathitrust.org/cgi/pt?id=hvd.hn5gdg&seq=11')

[davdat]

With 3.8.0, the request now works but the speed is significantly slower than 3.1.0, also a second request takes almost twice the time.

cloudscraper version: 3.8.0

scraper = cloudscraper.create_scraper()

Scraper (Request 1):
scraper.get("https://babel.hathitrust.org/cgi/pt?id=hvd.hn5gdg&seq=11")
Status: 200. Length: 24855. Time: 6.035901069641113

Scraper, Repeat (Request 2):
scraper.get("https://babel.hathitrust.org/cgi/pt?id=hvd.hn5gdg&seq=11")
Status: 200. Length: 24855. Time: 11.384987354278564

Turning on compatibility_mode results in failure:

scraper = cloudscraper.create_scraper(compatibility_mode=True)

Scraper (Request 1):
scraper.get("https://babel.hathitrust.org/cgi/pt?id=hvd.hn5gdg&seq=11")
Status: 403. Length: 7986. Time: 0.4248466491699219

Scraper, Repeat (Request 2):
scraper.get("https://babel.hathitrust.org/cgi/pt?id=hvd.hn5gdg&seq=11")
Status: 403. Length: 8157. Time: 1.6114177703857422

[zinzied]

Compatibility Mode Fixed! 🎉

Test Result: Status 200 in 4.79 seconds

Root Causes Fixed:

enable_intelligent_challenges=True caused recursion loops - removed
rotate_tls_ciphers=False prevented proper bypass - kept as default (True)
Fixed NoneType error in ml_optimization.py when timing_orchestrator is None
Final Compatibility Mode Configuration:

Disables only: enable_metrics, enable_performance_monitoring
Enables all bypass-critical features
Works identically to default mode now ✅
enhanced the speed too with new version 3.8.1

[davdat]

Works well enough, speed is still 2 times 3.1.0. Technically fixed :)

[zinzied]

So the issue now is about speed right ? Not something elses ?

[davdat]

Correct, the only issue is now speed.

[zinzied]

Problem-solving can wait… it’s fundraising time, right? 😁👌🤑

[zinzied]

you must close this issue request if solved then Thank you !