NelsaHost

Hardware AI Models Compatibility Compare

search Login

Can I run Qwen2-VL 72B on NVIDIA RTX 4060 Ti 8GB?

cancel

Fail/OOM

This GPU doesn't have enough VRAM

GPU VRAM

8.0GB

Required

144.0GB

Headroom

-136.0GB

VRAM Usage

0GB 100% used 8.0GB

info Technical Analysis

NVIDIA RTX 4060 Ti 8GB cannot run Qwen2-VL 72B in this configuration. The model requires 144.0GB but only 8.0GB is available, leaving you 136.0GB short.

lightbulb Recommendation

Consider using a more aggressive quantization (Q4_K_M, Q3_K_M) to reduce VRAM requirements, or upgrade to a GPU with more VRAM. Cloud GPU services like RunPod or Vast.ai offer affordable options.

tune Recommended Settings

Batch_Size

None

Context_Length

None

Inference_Framework

llama.cpp or vLLM

help Frequently Asked Questions

Can I run Qwen2-VL 72B on NVIDIA RTX 4060 Ti 8GB? expand_more

NVIDIA RTX 4060 Ti 8GB (8.0GB VRAM) cannot run Qwen2-VL 72B (72.00B) which requires 144.0GB. You are 136.0GB short. Consider using a more aggressive quantization (like Q4_K_M or Q3_K_M) or upgrading to a GPU with more VRAM.

How much VRAM does Qwen2-VL 72B need? expand_more

Qwen2-VL 72B requires approximately 144.0GB of VRAM.

What performance can I expect? expand_more

Estimated None tokens per second.