Running vLLM on Tesla V100 GPUs — Making Old Silicon Serve Modern LLMs

Introduction

Not everyone has access to an H100 cluster. In many parts of the world — Southeast Asia, Eastern Europe, Latin America, parts of Africa — compute power is scarce and expensive. Cloud GPU instances cost a premium when the nearest data center is an ocean away. On-premise options are limited to whatever the local reseller has in stock, which often means previous-generation hardware at best.

Yet the demand for LLM inference is everywhere. Developers need coding assistants. Startups need chatbots. Researchers need to experiment with open-weight models. The gap between “what people need” and “what people can afford” is enormous.

Running inference on CPU is technically possible, but painfully slow — expect single-digit tokens per second on a 32B parameter model. Consumer-grade GPUs like the RTX 4090 have impressive specs on paper, but 24 GB of VRAM simply cannot hold a 32B model at reasonable quantization levels without aggressive offloading that kills throughput. And even if you could afford multiple RTX cards, NVLink isn’t available on consumer hardware, so multi-GPU communication bottlenecks everything.

This article documents my experience getting vLLM running on 8× Tesla V100 GPUs (32 GB each, 256 GB total VRAM) to serve Qwen3-Coder-32B-A3B-Instruct — a model roughly equivalent to Claude Sonnet 4.5 in coding tasks. It wasn’t straightforward, but it works.

Assumptions

Let me paint the scenario. Imagine a junior engineer who just graduated from university. They’re based in a country where an A100 node costs more than their annual salary. They browse the second-hand server market and find a used node with 8× Tesla V100 SXM2 32 GB GPUs at a price they can barely justify to their bank account. NVLink interconnect is included. The total VRAM is 256 GB — more than enough to hold a 32B model with comfortable overhead for KV cache.

The model choice is deliberate. Qwen3-Coder-32B-A3B-Instruct is a Mixture-of-Experts (MoE) architecture with 32 billion total parameters but only ~3 billion active parameters per forward pass. This makes it remarkably efficient — you get 32B-class reasoning quality with inference costs closer to a 3B model. For coding tasks, it competes with frontier commercial models while being fully open-weight.

Here’s the hardware profile:

Component	Spec
GPU	8× Tesla V100 SXM2 32 GB
Compute Capability	7.0 (Volta architecture)
Interconnect	NVLink 2.0 (300 GB/s bi-directional per GPU pair)
Total VRAM	256 GB
Precision	FP16, FP32 (no BF16, no FP8)

The critical constraint: compute capability 7.0. The V100 was released in 2017. It predates BF16 support (Ampere, cc 8.0), FP8 (Hopper, cc 9.0), and many of the fused attention kernels that modern inference frameworks assume are available. This single number — 7.0 — is the source of nearly every problem that follows.

Deployment

The PyTorch Wall

The first thing you’ll discover is that PyTorch 2.8.x and later have dropped support for compute capability 7.0 in their pre-built binaries. If you install the latest PyTorch from pip, CUDA kernels silently fail or throw cryptic errors like:

CUDA error: no kernel image is available for execution on the device

This is because the PyTorch wheels ship only with kernels compiled for cc 7.5+ (or even 8.0+ in some builds). Your V100 is literally not invited to the party.

The cascade effect is brutal: vLLM, SGLang, and most Python-based inference frameworks depend on recent PyTorch versions. When PyTorch doesn’t support your hardware, nothing built on top of it does either.

Pinning the Right Software Stack

After significant trial and error, I found a software stack that works. The key insight is that you need to lock specific versions of three components simultaneously — the NVIDIA driver, the CUDA toolkit, and vLLM itself.

Component	Version	Why
NVIDIA Driver	575.57.x	Last driver series with robust V100 support and CUDA 12.8+ compatibility
CUDA Toolkit	12.8.x – 12.9.x	Required by vLLM’s CUDA kernels; 12.8 still includes cc 7.0 targets
vLLM	v0.9.2 or v0.10.1	V1 engine works on cc 7.0; must be compiled from source
PyTorch	2.6.x – 2.7.x	Last versions with cc 7.0 support in source builds

The driver version matters more than you’d think. Too old and you can’t use CUDA 12.8. Too new and NVIDIA may have deprecated V100 codepaths. The 575.57.x series sits in the sweet spot.

Compiling vLLM from Source

The pre-built vLLM wheels on PyPI do not include cc 7.0 kernels. You must compile from source with the correct TORCH_CUDA_ARCH_LIST:

# Clone the specific vLLM version
git clone --branch v0.10.1 https://github.com/vllm-project/vllm.git
cd vllm

# Set the compute capability target
export TORCH_CUDA_ARCH_LIST="7.0"
export MAX_JOBS=8

# Install build dependencies
pip install -r requirements/build.txt

# Build and install — this takes 30-60 minutes on a decent CPU
pip install -e .

A few things that will trip you up:

• TORCH_CUDA_ARCH_LIST="7.0" is critical. Without it, the build targets whatever your PyTorch was compiled for, which won’t include 7.0
• MAX_JOBS controls parallel compilation. Set it to your CPU core count, but watch memory — each compilation job can use 2-4 GB of RAM
• The build will emit hundreds of warnings about deprecated CUDA features. Ignore them — they’re warnings, not errors
• If the build fails with OOM, reduce MAX_JOBS

I have successfully compiled and tested both v0.9.2 and v0.10.1. Both work with the V1 engine on V100 hardware.

Running the Model

Once vLLM is compiled, serving the model is straightforward:

vllm serve Qwen/Qwen3-Coder-32B-A3B-Instruct \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --dtype float16 \
  --trust-remote-code

Key flags:

• --tensor-parallel-size 8 — distribute the model across all 8 V100s via NVLink
• --max-model-len 32768 — context window (more on this in Lessons Learned)
• --dtype float16 — V100 has no BF16 support; FP16 is your only option
• --gpu-memory-utilization 0.90 — leave 10% headroom for KV cache growth spikes

The model loads, the health check passes, and you can hit the OpenAI-compatible API at http://localhost:8000/v1/chat/completions. It works.

Lessons Learned

Context Window Trade-offs Are Real

This is the most important lesson: inference speed degrades significantly when the context window exceeds 128K tokens on V100 hardware.

The V100’s memory bandwidth (900 GB/s) and lack of hardware-accelerated attention mechanisms (no FlashAttention-2 native support, no PagedAttention hardware optimization) mean that KV cache management becomes the bottleneck at large context sizes. Below 128K tokens, throughput is acceptable — roughly 15-25 tokens/second for a single concurrent request. Above 128K, it drops to single digits and latency becomes impractical for interactive use.

My recommendation: cap --max-model-len at 32768 or 65536 for production workloads. If you need the full 128K+ context, batch your requests during off-peak hours and accept the slower throughput as a trade-off.

Context Length	Approx. Throughput (tok/s)	Practical?
≤ 32K	20–25	Yes — interactive use
32K – 64K	15–20	Yes — acceptable latency
64K – 128K	8–15	Marginal — batch only
> 128K	< 8	No — impractical for most uses

FP16 Requires Careful Attention

Without BF16 support, you’re stuck on FP16 for all inference. FP16 has a narrower dynamic range (±65,504 vs ±3.4×10³⁸ for BF16/FP32), which means numerical stability issues can surface on certain model architectures. With Qwen3-Coder, I haven’t observed accuracy degradation on coding tasks, but monitor your outputs — especially for math-heavy prompts.

The V1 Engine Is Your Only Option

vLLM has been migrating from the V0 engine to V1. On V100 hardware, the V0 engine has numerous compatibility issues with older CUDA compute capabilities. The V1 engine, while still maturing, handles cc 7.0 more gracefully. If you encounter errors about missing kernels or unsupported operations, make sure you’re explicitly using V1.

Build Reproducibility Matters

Document your exact software versions. A minor CUDA point release or a PyTorch patch can break the build. I recommend creating a Docker image once you have a working build so you never have to fight the compilation again:

# Save your working environment
pip freeze > requirements-v100-vllm.txt

# Tag your CUDA/driver versions in a Dockerfile or README
echo "NVIDIA Driver: $(nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1)"
echo "CUDA: $(nvcc --version | grep release)"
echo "PyTorch: $(python -c 'import torch; print(torch.__version__)')"
echo "vLLM: $(python -c 'import vllm; print(vllm.__version__)')"

Future Discussion

The Second-Hand GPU Market Is Becoming Strategic

As hyperscalers upgrade to H100/B200 clusters, a wave of decommissioned V100 and A100 hardware is hitting the secondary market. For organizations in compute-scarce regions, this creates a window of opportunity — serious inference capability at a fraction of the original cost. The question isn’t whether old GPUs can run LLMs. The question is how long the software ecosystem will continue supporting them.

The Software Support Cliff

The pattern is clear: NVIDIA and PyTorch are progressively dropping support for older compute capabilities. Today it’s cc 7.0 (V100). Tomorrow it may be cc 7.5 (T4, RTX 2080), and eventually cc 8.0 (A100). Each deprecation cycle leaves hardware stranded — not because the silicon can’t do the math, but because nobody compiles the kernels for it anymore. Community-maintained builds and forks will become increasingly important for extending the useful life of older hardware.

Mixture-of-Experts Models Change the Equation

The rise of MoE architectures like Qwen3-Coder-32B-A3B (32B total, 3B active) is a game-changer for older hardware. You get large-model quality with small-model inference costs. As more MoE models are released, the minimum hardware bar for useful LLM inference drops considerably. A V100 node that struggles with a dense 70B model can comfortably serve a 32B MoE model — the active parameter count, not the total, determines inference performance.

Distillation and Quantization Are Closing the Gap

Techniques like GPTQ, AWQ, and knowledge distillation continue to improve. A well-quantized 32B model on V100 hardware today delivers output quality that would have required a 70B model on an A100 a year ago. This trend favors older hardware — each generation of quantization and distillation research effectively extends the useful life of existing GPUs.

The Real Barrier Is Knowledge, Not Hardware

The biggest obstacle to running LLMs on older GPUs isn’t the hardware — it’s knowing which exact combination of driver, CUDA, PyTorch, and framework versions will work together. This article exists because I spent days figuring it out through trial and error. The AI community needs better documentation of these compatibility matrices, especially for hardware that’s no longer in the “officially supported” spotlight.

Key Takeaways

1. Tesla V100 GPUs can serve modern 32B-class LLMs — but only with careful version pinning and source compilation
2. Lock your stack: NVIDIA driver 575.57.x + CUDA 12.8.x–12.9.x + vLLM compiled from source with TORCH_CUDA_ARCH_LIST="7.0"
3. Cap your context window at 32K–64K for interactive use; above 128K, throughput degrades to impractical levels
4. MoE models are ideal for older hardware — Qwen3-Coder-32B-A3B gives 32B quality at 3B inference costs
5. Document and containerize your working build immediately — reproducibility is fragile on deprecated hardware