Skip to content

Add URL scraping for automatic context ingestion #4

@l2dnjsrud

Description

@l2dnjsrud

Description

Instead of manually pasting context text, users should be able to provide URLs (news articles, competitor pages, fan community posts) and have PhantomCrowd automatically scrape and ingest them.

From the roadmap

Listed in README roadmap as a planned feature.

Proposed approach

  • Add a URL input field in the campaign creation form
  • Backend scrapes the URL content (using httpx + BeautifulSoup or similar)
  • Extracted text is appended to context_text
  • Support multiple URLs

Difficulty

Intermediate. Backend scraping + frontend UI.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions