feat: add hnsw-rabitq support by egolearner · Pull Request #69 · alibaba/zvec

egolearner · 2026-02-04T11:47:19Z

resolve #42

egolearner · 2026-02-05T02:17:11Z

Depends on VectorDB-NTU/RaBitQ-Library#36

python/tests/test_collection_hnsw_rabitq.py

Cuiyus · 2026-02-27T02:20:21Z

@greptile

greptile-apps · 2026-02-27T02:23:55Z

Greptile Summary

This PR adds comprehensive HNSW-RaBitQ support to the zvec project, implementing a vector search index that combines Hierarchical Navigable Small World (HNSW) graph structure with RaBitQ quantization for memory-efficient approximate nearest neighbor search.

Major changes:

New HNSW-RaBitQ algorithm implementation with builder, searcher, and streamer components
RaBitQ vector quantization with configurable bits (default 7-bit) and k-means clustering
Integration with existing zvec framework including Python bindings and protocol buffers
Comprehensive test coverage at both C++ and Python levels
Added RaBitQ-Library as git submodule dependency

Issues found:

Variable-length arrays (VLAs) used in algorithm files are not standard C++ and will cause compilation failures on MSVC
Typo in .gitmodules with duplicate "thirdparty/" prefix in submodule name

Confidence Score: 3/5

This PR requires fixes for compilation issues before merging
The implementation is comprehensive with good test coverage, but contains critical syntax issues (VLAs) that will prevent compilation on MSVC and a configuration error in git submodules. Once these are fixed, the code appears well-structured
Pay close attention to src/core/algorithm/hnsw-rabitq/hnsw_rabitq_algorithm.cc and hnsw_rabitq_query_algorithm.cc which have VLA issues, and .gitmodules with the submodule name typo

Important Files Changed

Filename	Overview
.gitmodules	Added RaBitQ-Library submodule with typo in name (duplicate thirdparty/)
src/core/algorithm/hnsw-rabitq/hnsw_rabitq_algorithm.cc	Implements HNSW graph algorithm with node insertion and search; contains multiple VLAs (non-standard C++)
src/core/algorithm/hnsw-rabitq/hnsw_rabitq_query_algorithm.cc	Query algorithm implementation with VLA that needs replacement with std::vector
src/core/algorithm/hnsw-rabitq/hnsw_rabitq_builder.cc	Builder implementation with proper parameter validation and initialization
src/core/algorithm/hnsw-rabitq/rabitq_converter.cc	RaBitQ converter with k-means clustering for vector quantization training
src/include/zvec/db/index_params.h	Added HnswRabitqIndexParams class with proper integration into type system
python/tests/test_collection_hnsw_rabitq.py	Comprehensive Python tests for HNSW-RaBitQ collection operations
src/core/algorithm/hnsw-rabitq/hnsw_rabitq_searcher.cc	Search implementation with proper parameter validation and reformer initialization

Class Diagram

%%{init: {'theme': 'neutral'}}%%
classDiagram
    class HnswRabitqIndex {
        +build()
        +search()
        +stream()
    }
    
    class HnswRabitqBuilder {
        +init()
        +train()
        +build()
        -rabitq_converter_
    }
    
    class HnswRabitqSearcher {
        +init()
        +search()
        -entity_
        -reformer_
    }
    
    class HnswRabitqStreamer {
        +init()
        +add()
        +search()
        -entity_
        -reformer_
    }
    
    class HnswRabitqAlgorithm {
        +add_node()
        +search()
        -entity_
    }
    
    class HnswRabitqEntity {
        +get_neighbors()
        +update_neighbors()
        -graph_structure_
    }
    
    class RabitqConverter {
        +train()
        +transform()
        +to_reformer()
        -rotator_
        -centroids_
    }
    
    class RabitqReformer {
        +reform()
        +get_quantized()
        -rotator_
        -centroids_
    }
    
    class HnswRabitqContext {
        +dist_calculator()
        +visit_filter()
    }
    
    HnswRabitqIndex --> HnswRabitqBuilder
    HnswRabitqIndex --> HnswRabitqSearcher
    HnswRabitqIndex --> HnswRabitqStreamer
    
    HnswRabitqBuilder --> RabitqConverter
    HnswRabitqBuilder --> HnswRabitqAlgorithm
    HnswRabitqBuilder --> HnswRabitqEntity
    
    HnswRabitqSearcher --> HnswRabitqAlgorithm
    HnswRabitqSearcher --> HnswRabitqEntity
    HnswRabitqSearcher --> RabitqReformer
    
    HnswRabitqStreamer --> HnswRabitqAlgorithm
    HnswRabitqStreamer --> HnswRabitqEntity
    HnswRabitqStreamer --> RabitqReformer
    
    HnswRabitqAlgorithm --> HnswRabitqEntity
    HnswRabitqAlgorithm --> HnswRabitqContext
    
    RabitqConverter --> RabitqReformer

_{Last reviewed commit: 42a30d6}

greptile-apps

_{93 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-27T02:23:59Z

.gitmodules

 	path = thirdparty/magic_enum/magic_enum-0.9.7
 	url = https://github.com/Neargye/magic_enum.git
 	ignore = all
+[submodule "thirdparty/thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]


Submodule name has duplicate thirdparty/ prefix

Suggested change

[submodule "thirdparty/thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]

[submodule "thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]

please check and handle this.

You're right, the submodule name has a duplicate thirdparty/ prefix. The path is correct but the submodule name itself is wrong. Here's the fix:

Suggested change

[submodule "thirdparty/thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]

[submodule "thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]

This will match the pattern used by all other submodules in the file and correctly remove the duplicate prefix.

greptile-apps · 2026-02-27T02:23:59Z

src/core/algorithm/hnsw_rabitq/hnsw_rabitq_algorithm.cc

+    float dists[size];
+    const void *neighbor_vecs[size];


Variable-length arrays (VLAs) are not standard C++ and won't compile with MSVC. Use std::vector instead

Suggested change

float dists[size];

const void *neighbor_vecs[size];

std::vector<float> dists(size);

std::vector<const void*> neighbor_vecs(size);

please check and handle this.

Based on the provided context, the issue is clear. Lines 139-140 use variable-length arrays (VLAs) which aren't standard C++ and won't compile on MSVC. Here's the fix:

Suggested change

float dists[size];

const void *neighbor_vecs[size];

std::vector<float> dists(size);

std::vector<const void*> neighbor_vecs(size);

This replaces the VLAs with std::vector, making the code portable across all C++ compilers including MSVC.

greptile-apps · 2026-02-27T02:24:00Z

src/core/algorithm/hnsw_rabitq/hnsw_rabitq_algorithm.cc

+      (*ctx->mutable_stats_get_neighbors())++;
+    }
+
+    node_id_t neighbor_ids[neighbors.size()];


VLA not standard C++, won't compile with MSVC

Suggested change

node_id_t neighbor_ids[neighbors.size()];

std::vector<node_id_t> neighbor_ids(neighbors.size());

please check and handle this.

You're right, this is a VLA which isn't standard C++. The fix is to use std::vector:

Suggested change

node_id_t neighbor_ids[neighbors.size()];

std::vector<node_id_t> neighbor_ids(neighbors.size());

This will compile on MSVC and all standard C++ compilers while maintaining the same functionality.

greptile-apps · 2026-02-27T02:24:01Z

src/core/algorithm/hnsw_rabitq/hnsw_rabitq_algorithm.cc

+    float dists[size];
+    const void *neighbor_vecs[size];


VLAs not standard C++, won't compile with MSVC

Suggested change

float dists[size];

const void *neighbor_vecs[size];

std::vector<float> dists(size);

std::vector<const void*> neighbor_vecs(size);

please check and handle this.

You're right, those are VLAs which won't compile on MSVC. Replace them with std::vector:

Suggested change

float dists[size];

const void *neighbor_vecs[size];

std::vector<float> dists(size);

std::vector<const void*> neighbor_vecs(size);

This is standard C++ and will work across all compilers. You'll also need to update the references to use .data() when passing to batch_dist:

dc.batch_dist(neighbor_vecs.data(), size, dists.data());

greptile-apps · 2026-02-27T02:24:02Z

src/core/algorithm/hnsw_rabitq/hnsw_rabitq_query_algorithm.cc

+      (*ctx->mutable_stats_get_neighbors())++;
+    }
+
+    node_id_t neighbor_ids[neighbors.size()];


VLA not standard C++, won't compile with MSVC

Suggested change

node_id_t neighbor_ids[neighbors.size()];

std::vector<node_id_t> neighbor_ids(neighbors.size());

need to pay attention to these VLA issues.

feihongxu0824 · 2026-03-10T03:46:57Z

thirdparty/RaBitQ-Library/CMakeLists.txt

@@ -0,0 +1,4 @@
+add_library(rabitqlib INTERFACE)
+target_include_directories(
+    sparsehash INTERFACE "${CMAKE_CURRENT_SOURCE_DIR}/RaBitQ-Library-0.1/include"


sparsehash -> rabitqlib?

richyreachy · 2026-03-10T13:15:10Z

src/core/algorithm/hnsw_rabitq/rabitq_converter.cc

this class can be moved to quantizer folder with rabitq_reformer

richyreachy · 2026-03-11T02:48:12Z

src/core/algorithm/hnsw_rabitq/hnsw_rabitq_query_algorithm.cc

+
+void HnswRabitqQueryAlgorithm::expand_neighbors_by_group(
+    TopkHeap &topk, HnswRabitqContext *ctx) const {
+  // if (!ctx->group_by().is_valid()) {


to return with some message when group's setup?

implemented group by search, will test it after framework support group by .

richyreachy · 2026-03-11T02:51:36Z

src/core/algorithm/hnsw_rabitq/hnsw_rabitq_query_algorithm.cc

+          continue;
+        }
+      } else {
+        // Candidate cand{ResultRecord(candest.est_dist, candest.low_dist),


no need to check ex bits here? can it be checked earlier?

It's necessary to check ex_bits here.

first get_bin_est use single bit to estimate score

ex_bits > 0

yes. check if need to do full estimate with extra bits

no. Nothing to be done since we already used all the information.

richyreachy · 2026-03-11T02:54:06Z

src/core/algorithm/hnsw_rabitq/hnsw_rabitq_query_algorithm.cc

+
+      // TODO: check loop type
+
+      // if ((!topk.full()) || cur_dist < topk[0].second) {


lagacy codes here and elsewhere can be cleaned up.

richyreachy · 2026-03-11T03:03:45Z

src/core/algorithm/hnsw_rabitq/hnsw_rabitq_query_algorithm.cc

+  query_wrapper.set_g_add(q_to_centroids[cluster_id]);
+  float est_dist = rabitqlib::split_distance_boosting(
+      ex_data, ip_func_, query_wrapper, padded_dim_, ex_bits_, res.ip_x0_qr);
+  float low_dist = est_dist - (res.est_dist - res.low_dist) / (1 << ex_bits_);


can predefine param to convert to multiply manipulation:
double inv_divisor = 1.0 / (1 << ex_bits_);
auto result = (res.est_dist - res.low_dist) * inv_divisor;

get_ex_est is not used, deleted.

egolearner added the version/0.2.0 label Feb 4, 2026

egolearner marked this pull request as draft February 4, 2026 11:47

egolearner force-pushed the feat/rabitq branch from 3c02fb5 to 3381152 Compare February 4, 2026 12:02

feat: add hnsw-rabitq

0e5727a

egolearner force-pushed the feat/rabitq branch from 3381152 to 0e5727a Compare February 4, 2026 12:08

egolearner added 2 commits February 5, 2026 09:47

undefine transform

f3a4214

support config rotator type

b4794f6

egolearner added 19 commits February 5, 2026 11:26

support sample_count

676c457

fix ut

fd330c0

refactor: update interface

5977529

refactor: update interface

9713221

update rabitq index params

78127fd

fix interface update

af3f172

fix searcher test

2e153b4

fix streamer test

0427555

streamer support bf and add more ut

4c6f28a

rm env check

aac0ea3

add collection ut

d2809c3

add schema check

2f0ccbc

add rabitq query param binding

420d791

add integration test

32e2ed0

fix local_builder

733fa46

cleanup dist calculator

f381039

add RaBitQ-Library submodule

090913d

cleanup rabitq converter/reformer

afbf50c

Merge branch 'main' into feat/rabitq

5431adb

github-code-quality bot found potential problems Feb 10, 2026

View reviewed changes

python/tests/test_collection_hnsw_rabitq.py Fixed Show fixed Hide fixed

add files

3b06f74

egolearner marked this pull request as ready for review February 10, 2026 13:31

egolearner assigned richyreachy Feb 10, 2026

feihongxu0824 removed the version/0.2.0 label Feb 11, 2026

egolearner added 3 commits February 11, 2026 15:40

disable build on mac

822af9a

disable Feature_Optimize_HNSW_RABITQ

36aa0a7

disable python/tests/test_collection_hnsw_rabitq.py:13

86f84bd

github-code-quality bot found potential problems Feb 11, 2026

View reviewed changes

python/tests/test_collection_hnsw_rabitq.py Dismissed Show dismissed Hide dismissed

egolearner added 11 commits February 19, 2026 07:44

check avx2/avx512

df922f2

Merge remote-tracking branch 'origin/main' into feat/rabitq

91f706d

fix refine

80adf29

fix mac ci

f57c56e

Merge remote-tracking branch 'origin/main' into feat/rabitq

9dc8b64

add dimension check

56bbcc5

Merge remote-tracking branch 'origin/main' into feat/rabitq

04817ec

fix mac ci

f8713c5

Merge remote-tracking branch 'origin/main' into feat/rabitq

986e557

fix ci

f05a120

add missing lib

42a30d6

greptile-apps bot reviewed Feb 27, 2026

View reviewed changes

egolearner added 3 commits February 28, 2026 10:55

fix centroids selection for Cosine/InnerProduct

e5fb71b

rename hnsw-rabitq to hnsw_rabitq

8549247

Merge remote-tracking branch 'origin/main' into feat/rabitq

d93d32b

feihongxu0824 reviewed Mar 10, 2026

View reviewed changes

richyreachy reviewed Mar 10, 2026

View reviewed changes

richyreachy reviewed Mar 11, 2026

View reviewed changes

address comments

6570070

	[submodule "thirdparty/thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]
	[submodule "thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]

	node_id_t neighbor_ids[neighbors.size()];
	std::vector<node_id_t> neighbor_ids(neighbors.size());


		// TODO: check loop type

		// if ((!topk.full()) \|\| cur_dist < topk[0].second) {

Conversation

egolearner commented Feb 4, 2026

Uh oh!

egolearner commented Feb 5, 2026

Uh oh!

Uh oh!

Uh oh!

Cuiyus commented Feb 27, 2026

Uh oh!

greptile-apps bot commented Feb 27, 2026

Greptile Summary

Confidence Score: 3/5

Important Files Changed

Class Diagram

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richyreachy Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richyreachy Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

richyreachy Mar 10, 2026 •

edited

Loading

richyreachy Mar 11, 2026 •

edited

Loading