[Doc]: Does it works with single GPU? Or models like SXM2xPCIE?

[JoaoZaokk]

📚 The doc issue

Hi guys, im trying to build a homelab, but im going budget now, so im only able to buy one v100 16gb per month, maybe one month with the 4x sxm2 + one sxm2 16gb, but, now, not more than that. Im posting this bc v100 is such a great gpu, but if it wont work on single mode, A2 gets in the same price and has natural flashattn2, its weaker, but in the total process theyre alike (without flashattn).
Thanks!

Suggest a potential alternative/fix

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[titidatiti]

I have the same question. I'm using a Tesla V100 (32G) with PCIE converted from SXM2.
I've tried to build a docker image and run it.
However, I got the traceback showing there is an undefined symbol:

Traceback (most recent call last):
File "/usr/local/bin/vllm", line 4, in
from vllm.entrypoints.cli.main import main
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/init.py", line 3, in
from vllm.entrypoints.cli.benchmark.latency import BenchmarkLatencySubcommand
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/benchmark/latency.py", line 5, in
from vllm.benchmarks.latency import add_cli_args, main
File "/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/latency.py", line 16, in
from vllm.engine.arg_utils import EngineArgs
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 36, in
from vllm.config import (
File "/usr/local/lib/python3.12/dist-packages/vllm/config/init.py", line 5, in
from vllm.config.cache import CacheConfig
File "/usr/local/lib/python3.12/dist-packages/vllm/config/cache.py", line 13, in
from vllm.utils.mem_utils import format_gib, get_cpu_memory
File "/usr/local/lib/python3.12/dist-packages/vllm/utils/mem_utils.py", line 14, in
from vllm.platforms import current_platform
File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/init.py", line 252, in getattr
_current_platform = resolve_obj_by_qualname(platform_cls_qualname)()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils/import_utils.py", line 111, in resolve_obj_by_qualname
module = importlib.import_module(module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py", line 16, in
import vllm._C # noqa
^^^^^^^^^^^^^^
ImportError: /usr/local/lib/python3.12/dist-packages/vllm/_C.abi3.so: undefined symbol: _Z17awq_moe_gemm_sm70N2at6TensorES0_S0_S0_llll

[humanjesse]

@titidatiti the undefined symbol: _Z17awq_moe_gemm_sm70... means your _C.abi3.so doesn't have 1Cat's SM70 AWQ-MoE patch (the symbol comes from csrc/quantization/awq/gemm_sm70.cu) even though Python startup expects it. Rebuild from a clean checkout — cmake --build cmake-build-release --target _C -j N from inside the source tree — and confirm the resulting .so actually overwrites the one in site-packages.

@JoaoZaokk for single-GPU: Granite-4.1-8B-AWQ-INT4 just tested cleanly at TP=1 on a single V100 32GB — 6.74 GiB weights, ~91 tok/s decode, GPU 0 only. So 8B-class AWQ-INT4 works on one 32GB card. 16GB would be tight with that model since weights alone are ~6.7 GiB.

If useful as a docker layout reference, I keep a downstream-of-1Cat fork at https://github.com/humanjesse/vllm-v100 with GGUF support (Mistral-Small-4 119B Q4_K_M, MiMo-V2.5 310B Q3_K_M) verified on 1×, 4×, and 8× V100 SXM2 32GB.

[JoaoZaokk]

@titidatiti the undefined symbol: _Z17awq_moe_gemm_sm70... means your _C.abi3.so doesn't have 1Cat's SM70 AWQ-MoE patch (the symbol comes from csrc/quantization/awq/gemm_sm70.cu) even though Python startup expects it. Rebuild from a clean checkout — cmake --build cmake-build-release --target _C -j N from inside the source tree — and confirm the resulting .so actually overwrites the one in site-packages.

@JoaoZaokk for single-GPU: Granite-4.1-8B-AWQ-INT4 just tested cleanly at TP=1 on a single V100 32GB — 6.74 GiB weights, ~91 tok/s decode, GPU 0 only. So 8B-class AWQ-INT4 works on one 32GB card. 16GB would be tight with that model since weights alone are ~6.7 GiB.

If useful as a docker layout reference, I keep a downstream-of-1Cat fork at https://github.com/humanjesse/vllm-v100 with GGUF support (Mistral-Small-4 119B Q4_K_M, MiMo-V2.5 310B Q3_K_M) verified on 1×, 4×, and 8× V100 SXM2 32GB.

Now im deep in the LLM world to solve some problems, while my GPUs dont come, i already bought tree but it take time for it comes, im using mimo, but im really excited about the smx2, with mine 3080ti sometimes i receive 90-110ts, what i think its incredible, but when you say that sxm2 is something like that i am bit sad for that. But we'll see how itll works when it cames to me.

[NunoCruzSW]

I'm seeing a similar issue.

My setup uses 4 GPUs connected through PCIe adapters, but for testing I'm currently only using 2×16GB GPUs. I'm running on a Huananzhi F8D Plus motherboard.

The model appears to load successfully, but something seems to go wrong immediately afterward. The logs repeatedly show:

(EngineCore_DP0 pid=12317) INFO 05-23 03:14:15 [shm_broadcast.py:542] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/KV cache quantization).

(EngineCore_DP0 pid=12317) INFO 05-23 03:15:15 [shm_broadcast.py:542] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/KV cache quantization).

The message continues indefinitely, and the model never becomes available.

Has anyone experienced this with GPUs connected via PCIe adapters, or is there any additional debugging information I can provide?

[NunoCruzSW]

I solved the issue with NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce, but sure if NCCL_P2P_DISABLE is enough.

Wondering if there other settings to help on this issue