VLLM Memory Bug With Qwen-2.5-VL Models
Hey folks, we've got a heads-up about a potential hiccup with vLLM and the Qwen-2.5-VL family of models. It looks like there's a bug related to how torch calculates peak memory, and it's causing some wonky GPU memory utilization. Let's dive into the details and see what's what.
The Bug: Memory Miscalculation
The Core Issue: Memory Utilization Drop
So, here's the deal. We've noticed that when loading the Qwen-2.5-VL models (both the base and quantized versions) with vLLM, the GPU memory utilization isn't quite what it should be. We're seeing a significant drop in memory usage compared to previous versions of vLLM. Specifically, on a 4xL20 (or L40) setup, the numbers look like this:
- vLLM 0.10.2: ~45 GB out of 46 GB per GPU.
- vLLM 0.11.0: ~35 GB out of 45 GB per GPU.
This means that with vLLM 0.11.0, you're not getting as much bang for your buck in terms of GPU memory utilization. This in turn, reduces the supported concurrency.
Root Cause: Torch Peak Memory Calculation
The issue seems to stem from how vLLM calculates peak memory, specifically related to the torch library. It appears that the memory profiling isn't working as expected for these specific Qwen-2.5-VL models. This miscalculation leads to underutilization of the available GPU memory. The good news is, there's a workaround (more on that later!).
Affected Models: Qwen-2.5-VL Focus
It's important to note that this problem appears to be specific to the Qwen-2.5-VL family of models. We haven't seen the same issue with other models, such as the adept/fuyu-8b. This narrows down the scope of the problem and suggests it might be related to the specific architecture or characteristics of the Qwen-2.5-VL models.
Workaround and Solution
The --skip-mm-profile Flag
Fortunately, there's a quick fix! Using the --skip-mm-profile flag when launching vLLM brings the GPU memory utilization back in line with what we saw in vLLM 0.10.2. This flag essentially tells vLLM to skip the memory profiling step, which seems to be the source of the problem. While this is not a permanent solution, it does allow you to continue to use these models with good memory utilization.
Long-Term Fix: Addressing the Root Cause
The long-term solution involves pinpointing the exact cause of the memory calculation error within the vLLM code and the torch library. This may involve debugging the memory profiling code, checking for compatibility issues between vLLM and the specific Qwen-2.5-VL model architecture, or even reporting the issue to the torch team. The goal is to ensure accurate peak memory calculation without the need for workarounds.
Technical Details and Environment
System Configuration
Here's a snapshot of the environment where this bug was observed. This information is crucial for those who are digging into the code to reproduce and fix this bug.
(app-root) /opt/app-root$ vllm collect-env
/opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
DEBUG 11-06 14:38:18 [plugins/__init__.py:28] No plugins for group vllm.platform_plugins found.
DEBUG 11-06 14:38:18 [platforms/__init__.py:34] Checking if TPU platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:52] TPU platform is not available because: No module named 'libtpu'
DEBUG 11-06 14:38:18 [platforms/__init__.py:58] Checking if CUDA platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:78] Confirmed CUDA platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:106] Checking if ROCm platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:120] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 11-06 14:38:18 [platforms/__init__.py:127] Checking if XPU platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:146] XPU platform is not available because: No module named 'intel_extension_for_pytorch'
DEBUG 11-06 14:38:18 [platforms/__init__.py:153] Checking if CPU platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:58] Checking if CUDA platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:78] Confirmed CUDA platform is available.
INFO 11-06 14:38:18 [platforms/__init__.py:216] Automatically detected platform cuda.
DEBUG 11-06 14:38:22 [plugins/__init__.py:36] Available plugins for group vllm.general_plugins:
DEBUG 11-06 14:38:22 [plugins/__init__.py:38] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
DEBUG 11-06 14:38:22 [plugins/__init__.py:41] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
Collecting environment information...
==============================
System Info
==============================
OS : Red Hat Enterprise Linux 9.6 (Plow) (x86_64)
GCC version : (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5)
Clang version : Could not collect
CMake version : Could not collect
Libc version : glibc-2.34
==============================
PyTorch Info
==============================
PyTorch version : 2.8.0
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.12.9 (main, Aug 14 2025, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] (64-bit runtime)
Python platform : Linux-6.1.155-176.282.amzn2023.x86_64-x86_64-with-glibc2.34
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : 12.8.93
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration :
GPU 0: NVIDIA L40S
GPU 1: NVIDIA L40S
GPU 2: NVIDIA L40S
GPU 3: NVIDIA L40S
Nvidia driver version : 580.95.05
cuDNN version : Probably one of the following:
/usr/lib64/libcudnn.so.9.13.1
/usr/lib64/libcudnn_adv.so.9.13.1
/usr/lib64/libcudnn_cnn.so.9.13.1
/usr/lib64/libcudnn_engines_precompiled.so.9.13.1
/usr/lib64/libcudnn_engines_runtime_compiled.so.9.13.1
/usr/lib64/libcudnn_graph.so.9.13.1
/usr/lib64/libcudnn_heuristic.so.9.13.1
/usr/lib64/libcudnn_ops.so.9.13.1
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True
==============================
CPU Info
==============================
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7R13 Processor
CPU family: 25
Model: 1
Thread(s) per core: 2
Core(s) per socket: 24
Socket(s): 1
Stepping: 1
BogoMIPS: 5299.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 768 KiB (24 instances)
L1i cache: 768 KiB (24 instances)
L2 cache: 12 MiB (24 instances)
L3 cache: 96 MiB (3 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsa: Mitigation; Clear CPU buffers
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.3.1
[pip3] numpy==2.2.6
[pip3] nvidia-cudnn-frontend==1.15.0
[pip3] nvidia-ml-py==13.580.82
[pip3] pynvml==13.0.1
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0
[pip3] torchaudio==2.8.0
[pip3] torchvision==0.23.0
[pip3] transformers==4.57.1
[pip3] triton==3.4.0
[conda] Could not collect
==============================
vLLM Info
==============================
ROCM Version : Could not collect
vLLM Version : 0.11.0+rhai1
vLLM Build Flags:
CUDA Archs: ; ROCm: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NODE NODE NODE 0-47 0 N/A
GPU1 NODE X NODE NODE 0-47 0 N/A
GPU2 NODE NODE X NODE 0-47 0 N/A
GPU3 NODE NODE NODE X 0-47 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
==============================
Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.0 driver>=525.60.13
CUDA_CACHE_DISABLE=1
TORCH_CUDA_ARCH_LIST=
NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
VLLM_WORKER_MULTIPROC_METHOD=spawn
VLLM_USAGE_SOURCE=rhaiis-official-build
CUDA_VERSION=12.8.1
TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=15
TORCH_NCCL_DUMP_ON_TIMEOUT=0
VLLM_USAGE_STATS_SERVER=https://console.redhat.com/api/rhaiis-stats
VLLM_HAS_FLASHINFER_CUBIN=true
VLLM_NO_USAGE_STATS=1
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
VLLM_LOGGING_LEVEL=DEBUG
CUDA_MODULE_LOADING=LAZY
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_default
Key Libraries and Versions
- PyTorch: 2.8.0
- vLLM: 0.11.0+rhai1
- CUDA: 12.8
This information will be super helpful for anyone looking to investigate the issue or replicate the bug.
Conclusion: Keeping an Eye on Memory
So, there you have it, folks! We've identified a memory utilization issue with vLLM and Qwen-2.5-VL models. We've got a temporary fix, and we're looking into a long-term solution. Keep an eye on the vLLM updates and the official documentation for further news on this issue. Thanks for tuning in, and happy coding!