Segfault at process exit when zvec is statically linked (glog/protobuf destructor ordering)

[thedonmon]

Description

When zvec is statically linked into a host application (in our case via a Rust/Elixir NIF bridge), the process segfaults during shutdown. All zvec operations work correctly during the process lifetime — the crash only occurs at exit.

Root Cause

The segfault is caused by C++ static destructor ordering. zvec statically links glog, protobuf, gflags, and RocksDB, all of which register atexit handlers for global cleanup. At process exit, these destructors run in reverse registration order, but some depend on globals that were already destroyed by earlier destructors (e.g., glog tries to flush to a logger after protobuf has cleaned up).

Steps to Reproduce

Statically link zvec (with all dependencies) into any application, perform some operations, then exit. The process will segfault with exit code 139 (SIGSEGV).

Impact

Low — this is cosmetic. All operations work correctly. The crash only happens at process shutdown. In long-running applications (servers, daemons), this is never visible. It primarily affects test runners that start and stop the process.

Suggested Fixes

• Provide an explicit zvec::Shutdown() function that users can call before exit to clean up glog/protobuf/gflags in the correct order
• Or use __attribute__((destructor)) with explicit priority ordering for cleanup functions
• Or guard the glog/protobuf shutdown code against double-free / use-after-free of globals

Environment

• macOS ARM64 (Apple Silicon) + Linux x86_64
• zvec built with -DBUILD_SHARED_LIBS=OFF (static linking)
• Host: Elixir/Rust NIF via Rustler + cxx bridge
• Reproducible with any host that statically links zvec and then exits

[egolearner]

Could you provide a minimal reproducible example? I've tried building the C++ and C API examples with static linking, but the segmentation fault does not occur on my end.

[thedonmon]

Thanks for looking into this @egolearner! Here's a minimal reproduction:

Minimal C++ Example

// minimal_segfault.cc
#include <zvec/db/collection.h>
#include <zvec/db/schema.h>

using namespace zvec;

int main() {
    // Create a minimal schema with one vector field
    auto schema = std::make_shared<CollectionSchema>("test");
    schema->add_field(std::make_shared<FieldSchema>(
        "vec", DataType::VECTOR_FP32, 4, false,
        std::make_shared<HnswIndexParams>(MetricType::IP)));

    // Create and open a collection — this initializes glog/protobuf statics
    std::string path = "./test_segfault";
    system(("rm -rf " + path).c_str());
    CollectionOptions options{false, true};
    auto result = Collection::CreateAndOpen(path, *schema, options);
    if (!result.has_value()) {
        return 1;
    }

    // Insert one document
    Doc doc;
    doc.set_pk("doc_1");
    doc.set<std::vector<float>>("vec", {1.0f, 2.0f, 3.0f, 4.0f});
    std::vector<Doc> docs{doc};
    result.value()->Insert(docs);

    // Exit — static destructors fire here and cause SIGSEGV
    return 0;
}

Build

The key requirement is static linking with -Wl,-force_load (macOS) or --whole-archive (Linux):

add_executable(minimal_segfault minimal_segfault.cc)
target_link_libraries(minimal_segfault PRIVATE zvec_db)

Using the same CMake config from examples/c++/CMakeLists.txt which applies -Wl,-force_load on macOS.

Environment

• macOS 15 (Sequoia), Apple Silicon (M4 Max)
• zvec built from source with -DBUILD_SHARED_LIBS=OFF
• Also reproduced on Linux x86_64

Stack trace (macOS, lldb)

* thread #1, stop reason = EXC_BAD_ACCESS (code=1, address=0x...)
    frame #0: libsystem_pthread.dylib`_pthread_mutex_lock
    frame #1: libglog.a`google::LogMessage::~LogMessage()
    ...

The crash is in glog's static cleanup — it tries to acquire a mutex on a logger object that was already destroyed by protobuf's ShutdownProtobufLibrary() running first (or vice versa). The ordering is non-deterministic because both register atexit handlers independently.

Note

This does not segfault when:

• Using shared/dynamic linking (.dylib / .so)
• Calling google::protobuf::ShutdownProtobufLibrary() and google::ShutdownGoogleLogging() manually before main() returns

It only segfaults with static linking where -force_load / --whole-archive pulls in all object files and their static initializers/destructors.

[thedonmon]

Update: After more thorough testing, I need to correct my previous comment.

Cannot reproduce with standalone C++ or macOS

I built and tested the following, all exit cleanly on macOS ARM64 (Apple Silicon):

1. Standalone C++ example (examples/c++/db/main.cc) — exits with code 0
2. Standalone Rust binary with cxx bridge to zvec, statically linked with -Wl,-force_load — exits with code 0
3. Minimal Elixir/Rust NIF (Rustler + cxx → zvec) with insert, optimize, flush, multiple collections — exits with code 0 across 10+ runs

Only reproduced in Linux Docker (CI)

The segfault (exit code 139) was only observed in our CI environment — Docker container on ubuntu-latest with GitHub Actions. The CI workflow at ex_zvec/.github/workflows/ci.yml had to accept exit code 139 as passing.

Since I cannot reliably reproduce this in a standalone minimal example outside of the Docker CI environment, and since we have since migrated to a pure Rust implementation (zvec-rs), we no longer experience this issue.

I'm going to close this as I can't provide a reliable standalone reproduction. It may have been specific to the Linux Docker environment, the particular glibc/libstdc++ versions, or the BEAM VM's signal handling interacting with the static destructors in a way that doesn't happen on macOS.

Apologies for the incomplete initial report — should have verified reproduction before filing.

[thedonmon]

Closing — unable to reproduce in a standalone minimal example. Only observed in Docker CI (ubuntu-latest). We've migrated to a pure Rust implementation so this is no longer an issue for us.

[thedonmon]

To add some context — we ran into this while building ex_zvec, an Elixir NIF bridge to zvec via Rustler + cxx. The segfault showed up in our Docker-based CI pipeline (ubuntu-latest) when the BEAM VM shut down after running tests. All tests passed, but the process exited with code 139.

We never saw it locally on macOS, and couldn't reproduce it in a standalone C++ or Rust binary — only through the NIF loaded into the Erlang VM inside Docker. It's possible the BEAM's signal handling or atexit sequence interacts differently with the static destructors on Linux.

We've since replaced the C++ bridge with a pure Rust reimplementation (zvec-rs), so this is no longer something we can actively debug. Appreciate you looking into it though!

[thedonmon]

Reopening — I dug into the CI logs more carefully and the picture is clearer now. There are actually two issues, both Linux-specific when zvec is loaded as a shared library (NIF) into the BEAM VM.

Issue 1: Static factory registration fails on Linux

When zvec is compiled into a .so NIF library (via Rustler/cxx) and loaded by the Erlang VM with dlopen, the static self-registration constructors don't fire properly — even with --whole-archive. The CI logs show:

[ERROR] Failed to create metric, name InnerProduct
[ERROR] Failed to create and init metric
[ERROR] Failed to init index
[ERROR] Failed to open vector indexer for field: FieldSchema{name:'embedding',...}

The InnerProduct metric factory was never registered because the static constructors in libzvec_core.a didn't execute when the NIF .so was loaded via dlopen. This works correctly on macOS with -force_load.

Issue 2: Segfault at BEAM process exit (code 139)

After the tests finish, the BEAM VM shuts down and dlcloses the NIF library, triggering C++ static destructors. These crash with SIGSEGV — the glog/protobuf/gflags destructor ordering issue from the original report.

How to reproduce

The reproduction path is:

1. Build zvec as static libraries on Linux
2. Link them into a shared library (.so) using --whole-archive
3. Load that .so via dlopen (this is what the Erlang VM does for NIFs)
4. Use zvec → factory registration fails silently
5. Exit → static destructors crash

This does not happen:

• On macOS (with -force_load)
• When the final binary is a standalone executable (not dlopened)
• The --whole-archive flag may not propagate correctly when the final output is a .so rather than an executable

Environment

• Linux x86_64 (Docker, ubuntu-latest in GitHub Actions)
• zvec built with --whole-archive linking
• NIF loaded by BEAM via dlopen

The CI logs showing both issues: https://github.com/thedonmon/ex_zvec/actions/runs/22839628294

We've migrated to a pure Rust reimplementation so this doesn't block us, but wanted to document the actual root cause properly in case it helps others.

[egolearner]

Hi @thedonmon , thanks for your effort and detailed information provided. We will dig into this later this week.

[egolearner]

Still not reproducible following your instructions.

1. Build zvec as static libraries on Linux
2. Link them into a shared library (.so) using --whole-archive
3. Create a test program, which dlopens libzvec.so and try to use zvec facilities.

#include <dlfcn.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>

/* 函数指针类型，与 dummy.cc 中 extern "C" 声明一致 */
typedef void  *(*zvec_create_metric_fn)(const char *metric_name,
                                        int         data_type,
                                        uint32_t    dimension);
typedef float  (*zvec_metric_compute_distance_fn)(void         *handle,
                                                  const float  *vec_a,
                                                  const float  *vec_b,
                                                  uint32_t      dim);
typedef void   (*zvec_list_metrics_fn)(void);
typedef void   (*zvec_destroy_metric_fn)(void *handle);

int main(void) {
  const char *library_path = "./libzvec.so";

  printf("Loading %s via dlopen...\n", library_path);

  /*
   * RTLD_LAZY     : 延迟解析符号，加载时不立即绑定所有符号
   * RTLD_DEEPBIND : 优先从库自身的符号表查找符号，隔离与主程序的符号冲突
   *
   * 注意：RTLD_DEEPBIND 必须与 RTLD_LAZY 或 RTLD_NOW 组合使用，
   *       单独传 RTLD_DEEPBIND 会导致 "invalid mode" 错误。
   */
  void *library_handle = dlopen(library_path, RTLD_LAZY | RTLD_DEEPBIND);
  if (library_handle == NULL) {
    fprintf(stderr, "dlopen failed: %s\n", dlerror());
    return EXIT_FAILURE;
  }
  printf("Successfully loaded %s\n", library_path);

  /* 通过 dlsym 获取封装函数 */
  zvec_list_metrics_fn             list_metrics =
      (zvec_list_metrics_fn)dlsym(library_handle, "zvec_list_metrics");
  zvec_create_metric_fn            create_metric =
      (zvec_create_metric_fn)dlsym(library_handle, "zvec_create_metric");
  zvec_metric_compute_distance_fn  compute_distance =
      (zvec_metric_compute_distance_fn)dlsym(library_handle,
                                             "zvec_metric_compute_distance");
  zvec_destroy_metric_fn           destroy_metric =
      (zvec_destroy_metric_fn)dlsym(library_handle, "zvec_destroy_metric");

  if (!list_metrics || !create_metric || !compute_distance || !destroy_metric) {
    fprintf(stderr, "dlsym failed: %s\n", dlerror());
    dlclose(library_handle);
    return EXIT_FAILURE;
  }

  /* 列出所有已注册的 metric */
  list_metrics();

  /* 创建 SquaredEuclidean metric，FP32（data_type=2），维度 4 */
  const char *metric_name = "SquaredEuclidean";
  int         data_type   = 2;   /* IndexMeta::DataType::DT_FP32 */
  uint32_t    dimension   = 4;

  printf("Creating metric: %s (data_type=%d, dim=%u)\n",
         metric_name, data_type, dimension);

  void *metric_handle = create_metric(metric_name, data_type, dimension);
  if (metric_handle == NULL) {
    fprintf(stderr, "create_metric failed\n");
    dlclose(library_handle);
    return EXIT_FAILURE;
  }
  printf("Metric created successfully\n");

  /* 计算两个向量之间的距离 */
  float vec_a[4] = {1.0f, 2.0f, 3.0f, 4.0f};
  float vec_b[4] = {4.0f, 3.0f, 2.0f, 1.0f};
  float distance = compute_distance(metric_handle, vec_a, vec_b, dimension);
  printf("Distance(%s) between [1,2,3,4] and [4,3,2,1] = %.4f\n",
         metric_name, distance);

  /* 销毁 metric */
  destroy_metric(metric_handle);
  printf("Metric destroyed\n");

  dlclose(library_handle);
  printf("Library closed, exiting.\n");

  return EXIT_SUCCESS;
}

result

Loading ./libzvec.so via dlopen...
Successfully loaded ./libzvec.so
[zvec] registered metrics: Cosine Euclidean Hamming InnerProduct InnerProductSparse MipsSquaredEuclidean MipsSquaredEuclideanSparse NormalizedCosine QuantizedInteger SquaredEuclidean SquaredEuclideanSparse
Creating metric: SquaredEuclidean (data_type=2, dim=4)
Metric created successfully
Distance(SquaredEuclidean) between [1,2,3,4] and [4,3,2,1] = 20.0000
Metric destroyed
Library closed, exiting.