Skip to content

[help-wanted] Scraper'lara retry mekanizması ve hata yönetimi ekle #9

@Metrohan

Description

@Metrohan

What to build / Ne yapılacak

EN: Scrapers (TechCareer, Youthall, Akbank, Pupilica, Kodluyoruz, Anbean, Coderspace) currently fail silently when they encounter network errors, timeouts, or unexpected markup. One broken scraper can bring down the whole scrape run. Each scraper should have standardized error handling, retry logic with exponential backoff, and structured logging so failures are visible without blocking other scrapers.

TR: Scraper'lar (TechCareer, Youthall, Akbank, Pupilica, Kodluyoruz, Anbean, Coderspace) şu an ağ hataları, timeout'lar veya beklenmedik markup ile karşılaştığında sessizce başarısız oluyor; bir scraper'ın çökmesi tüm scrape çalışmasını durdurabilir. Her scraper için standart try/except, exponential backoff ile retry mekanizması ve yapılandırılmış loglama eklenmeli; bir scraper çökse bile diğerleri çalışmaya devam etmeli.

Acceptance criteria

  • Each scraper is wrapped in a try/except that catches network and parsing errors without propagating to the orchestrator.
  • Failed requests are retried up to 3 times with exponential backoff before the scraper gives up.
  • On final failure, the scraper logs: scraper name, error type, and message (visible in ScraperLog).
  • A failure in one scraper does not prevent other scrapers from running.
  • Unit tests verify that a simulated network error triggers retry and logs correctly.

Blocked by

None — can start immediately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp-wantedExtra attention is neededscraperRelated to scraper modules

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions