基于 Transformer + FAISS 的高性能语义文本去重工具，面向大规模语料库
-
Updated
Aug 12, 2025 - Python
#
text-deduplication
Here are 4 public repositories matching this topic...
Tool to deduplicate file contents
-
Updated
Jan 26, 2026 - Go
high-performance website content extractor
data-extraction content-analysis content-aggregation website-scraping competitive-research site-structure-analysis bulk-url-crawler text-deduplication parallel-crawling web-data-insights
-
Updated
Nov 11, 2025 - Python
A hyper-optimized, ultra-low-latency real-time text deduplication engine using C++20, AVX2 SIMD vectorization, MinHash, and Locality-Sensitive Hashing (LSH) for sub-millisecond duplicate checks on large-scale streams.
redis reddit-api lsh minhash locality-sensitive-hashing low-latency avx2 smd cpp20 moderation-tool developer-sandbox text-deduplication
-
Updated
Jun 30, 2026 - JavaScript
Improve this page
Add a description, image, and links to the text-deduplication topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the text-deduplication topic, visit your repo's landing page and select "manage topics."