Dear Community and MedGemma Developer,
I'm seeking specific guidance on hardware recommendations, particularly regarding GPU configurations for optimal performance and VRAM utilization.
Here are my main questions:
MedGemma Model Version: We are interested in both the MedGemma 4B (multimodal) and MedGemma 27B (text) models. Could you confirm the specific hardware requirements (especially VRAM) for inferencing each of these models on-premise, assuming non-quantized versions if possible, or advise on recommended quantization levels for local deployment?
Multi-GPU Support (NVIDIA RTX 4090):
Our current consideration for the server involves using two NVIDIA RTX 4090 GPUs. These cards offer a combined 48 GB of VRAM.
Does MedGemma's on-premise implementation effectively support multi-GPU configurations for inference? Specifically, can the workload (data or model parallelism) be efficiently distributed across two RTX 4090 cards to maximize throughput or to run larger models/batches that wouldn't fit on a single GPU?
Are there any specific recommendations or known limitations when using consumer-grade multi-GPU setups like 2x RTX 4090 compared to professional-grade cards (e.g., NVIDIA RTX A6000 with NVLink)?
Overall Server Hardware Guidance:
Beyond the GPU, are there specific CPU, RAM, and SSD recommendations for a robust and performant MedGemma inference server, considering typical clinical workloads (e.g., volume of image/document uploads)?
Are there any best practices or pitfalls to avoid when setting up an on-premise environment for MedGemma in terms of software stack (e.g., specific Linux distributions, containerization strategies like Docker, or required NVIDIA CUDA/cuDNN versions)?
Our goal is to ensure a stable, secure, and high-performance local setup that complies with clinical data handling best practices. Any insights or pointers to relevant documentation would be greatly appreciated.
Thank you for your time and support!
Dear Community and MedGemma Developer,
I'm seeking specific guidance on hardware recommendations, particularly regarding GPU configurations for optimal performance and VRAM utilization.
Here are my main questions:
MedGemma Model Version: We are interested in both the MedGemma 4B (multimodal) and MedGemma 27B (text) models. Could you confirm the specific hardware requirements (especially VRAM) for inferencing each of these models on-premise, assuming non-quantized versions if possible, or advise on recommended quantization levels for local deployment?
Multi-GPU Support (NVIDIA RTX 4090):
Our current consideration for the server involves using two NVIDIA RTX 4090 GPUs. These cards offer a combined 48 GB of VRAM.
Does MedGemma's on-premise implementation effectively support multi-GPU configurations for inference? Specifically, can the workload (data or model parallelism) be efficiently distributed across two RTX 4090 cards to maximize throughput or to run larger models/batches that wouldn't fit on a single GPU?
Are there any specific recommendations or known limitations when using consumer-grade multi-GPU setups like 2x RTX 4090 compared to professional-grade cards (e.g., NVIDIA RTX A6000 with NVLink)?
Overall Server Hardware Guidance:
Beyond the GPU, are there specific CPU, RAM, and SSD recommendations for a robust and performant MedGemma inference server, considering typical clinical workloads (e.g., volume of image/document uploads)?
Are there any best practices or pitfalls to avoid when setting up an on-premise environment for MedGemma in terms of software stack (e.g., specific Linux distributions, containerization strategies like Docker, or required NVIDIA CUDA/cuDNN versions)?
Our goal is to ensure a stable, secure, and high-performance local setup that complies with clinical data handling best practices. Any insights or pointers to relevant documentation would be greatly appreciated.
Thank you for your time and support!