Skip to content

feat: update referer database with chatbot/AI category#5

Merged
stevan-borus merged 2 commits intomainfrom
fix/update-referer-database
Apr 2, 2026
Merged

feat: update referer database with chatbot/AI category#5
stevan-borus merged 2 commits intomainfrom
fix/update-referer-database

Conversation

@stevan-borus
Copy link
Copy Markdown
Owner

Summary

Updates the bundled referer database with 100+ new domains across all categories, most notably adding the entirely new chatbot/AI category.

New chatbot/AI category (23 sources)

ChatGPT, Claude, Perplexity, Google Gemini, Microsoft Copilot, DeepSeek, Grok, Phind, Mistral/Le Chat, Poe, Character.AI, Meta AI, Baidu AI, Chat+, ChatGLM, iAsk, Jasper, NotebookLM, Qwen, Writesonic, You, Pi, HuggingChat

New social sources

  • Mastodon (11 major instances)
  • Discord (discord.com, discordapp.com)
  • x.com added to Twitter
  • bsky.app added to Bluesky
  • Additional subdomains for TikTok, Facebook, Reddit

Updated from upstream

  • Snowplow referer-parser (master): +11 email, +3 social, +2 search sources
  • Matomo searchengine-and-social-list: additional AI assistants

Stats

Category Before After
chatbot 0 23 sources, 30 domains
social 87 92 sources (+Mastodon, Discord, domain additions)
search 236 237 sources
email 38 47 sources
paid 45 45 sources
unknown 3 3 sources

Test plan

  • 35 tests pass (24 existing + 11 new)
  • New tests cover: ChatGPT, Claude, Perplexity, Gemini, DeepSeek, Grok, Copilot, Phind, Bluesky, Mastodon, Brave Search
  • Build succeeds
  • All existing tests still pass (no regressions)

- Add chatbot/AI category: ChatGPT, Claude, Perplexity, Gemini, DeepSeek,
  Grok, Copilot, Phind, Mistral, Poe, Character.AI, Meta AI, and 10 more
- Add missing social: Mastodon (11 instances), Discord, x.com for Twitter,
  bsky.app for Bluesky, TikTok/Facebook/Reddit subdomains
- Update to Snowplow upstream: +11 email, +3 social, +2 search sources
- Update README to document chatbot medium and AI detection
- Sources: Snowplow referer-parser + Matomo searchengine-and-social-list
@stevan-borus stevan-borus merged commit 9a525d9 into main Apr 2, 2026
4 checks passed
@stevan-borus stevan-borus deleted the fix/update-referer-database branch April 2, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant