vLLM and Transformers Backend: Next-Gen LLM Inference at Native Speed

vLLM and Transformers Backend: Next-Gen LLM Inference at Native Speed

The biggest bottleneck when running generative AI models in production is inference speed. While the Transformers library supports over 450 architectures, vLLM delivers high throughput with continuous batching and custom attention kernels. Now these two forces are merging: Transformers models can run in vLLM with a single flag, achieving the same speed as hand-written implementations.

In this article, we'll examine the latest update to the Transformers modeling backend in vLLM, how it works, the Qwen3 benchmark results, and its impact on production deployments.

The Problem: Separate vLLM Implementations for Every Model

When a new model is released in the ML ecosystem, the community puts significant effort into making it work across different frameworks. After a model is added to Transformers, a separate implementation had to be written for vLLM with custom optimizations. This process could take weeks and maintenance costs grew with every update.

This situation was especially frustrating for model authors. Adding a model to Transformers was already a substantial task. Then rewriting it with custom optimizations for vLLM was required on top of that. Each framework maintained its own codebase, and bug fixes and improvements had to be synchronized.

In the old workflow shown above, the model author first integrates into Transformers, then writes a separate implementation for vLLM with custom optimizations. This meant both time and maintenance costs. In an ecosystem where an average of 3 new models are added every week, this cost compounds rapidly.

The Solution: Single Integration, Dual Performance

The Hugging Face team has improved the Transformers modeling backend in vLLM so much that any model added to Transformers can now be run directly in vLLM and deliver the same (or better) performance as hand-written implementations.

In the new workflow, the model is only added to Transformers and automatically runs at native speed in vLLM. The model author doesn't need to do anything extra. This means Transformers' 450+ architecture support automatically flows into the vLLM ecosystem.

History: The First Integration and Its Limitations

The Transformers and vLLM integration actually started last year. At that time, a bridge was built to run Transformers model implementations in vLLM. However, this bridge only optimized the attention mechanism. So Transformers models could run in vLLM but weren't as fast as hand-written implementations.

Model authors still wrote custom vLLM implementations when they wanted the best performance. This led to the perception that the Transformers backend was "good but not good enough." The new update completely changes that perception.

How Does It Work? torch.fx and AST Manipulation

This update is built on two Python technologies:

• torch.fx: PyTorch's graph analysis tool. It statically analyzes the model's computation graph and detects optimizable patterns. This tool can understand the model's structure before running it.
• AST (Abstract Syntax Tree): Python's abstract syntax tree. It's used to rewrite operations in place on the source code once patterns are identified. Source code manipulation preserves the original model structure but transforms it into an optimized version at runtime.

The system follows these steps:

1. The model's graph structure is analyzed with torch.fx and the computation flow is extracted.
2. Known optimization patterns (e.g., mergeable linear layers, MoE expert layers) are detected.
3. AST manipulates the source code and rewrites operations in place.
4. The transformed model becomes compatible with vLLM's custom kernels and moves to the compilation phase.

This process happens at runtime, during model loading. No manual changes to model code are needed. It's completely transparent from the user's perspective.

What Optimizations Are Applied?

Fused Operations

Fused operations matched with vLLM's (ultra) optimized kernels are applied. The most important ones are:

• MergedColumnParallelLinear: Merges multiple linear layers into a single parallel operation. This reduces GPU memory accesses and increases compute efficiency.
• QKVParallelLinear: Runs Query, Key, and Value projections as a single parallel block. It optimizes the most critical part of the attention mechanism.
• Expert Parallelization (EP): Custom kernels for expert parallelization in MoE models. Critical for large MoE models like the 235B parameter variant.

Parallelization Plans

Thanks to fused operations, the system can automatically derive tensor parallel (TP) and pipeline parallel (PP) plans. When the decoder block list is easily identifiable, pipeline parallel plans are also generated automatically. This eliminates the need for manual configuration in multi-GPU setups.

Compilation Compatibility

Transformed models remain fully compatible with torch.compile and CUDA Graphs. So whatever compilation optimizations hand-written vLLM models get, the Transformers backend gets them too. This can mean an additional 20-30% performance gain from compiled models.

Bidirectional Use

Transformers model implementations can also be used in training. The same model code supports training, evaluation, and RL rollouts. vLLM model implementations are inference-only. This means research teams can use a single codebase for both training and serving.

Benchmark Results: Qwen3 Comparison

The Hugging Face team used three different Qwen3 models to compare the Transformers backend against hand-written native implementations. Each model was tested under three conditions that are identical in every way except the code path:

• native: --model-impl vllm, vLLM's hand-written model (comparison baseline).
• after: --model-impl transformers with the PR (new backend).
• before: --model-impl transformers without the PR (old backend).

Models tested:

• Qwen3-4B: 4 billion parameter dense model on a single GPU. Close to edge deployment scenarios.
• Qwen3-32B: 32 billion parameter dense model with tensor parallelism across 2 GPUs. Mid-scale enterprise usage.
• Qwen3-235B-A22B-FP8: 235 billion parameter Mixture-of-Experts model with data + expert parallelism on 8xH100 node. Large-scale serving.

The results are clear: the Transformers modeling backend matches or beats native throughput on all three models. And this is achieved with a single --model-impl transformers flag. All test scripts are published as a Gist for reproducibility.

Usage: Getting Started with a Single Flag

To run any Hugging Face model with the Transformers backend, a single flag is enough:

# Upgrade the vllm pip package
uv pip install --upgrade vllm --torch-backend auto

# Serve with a single GPU
vllm serve Qwen/Qwen3-4B --model-impl transformers

For multi-GPU setups, parallelism options work the same way:

# Tensor parallel with 2 GPUs
vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2

# Data + expert parallel with 8 GPUs
vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers --data-parallel-size 8 --enable-expert-parallel

Nothing else changes in your serving setup. Your existing parallelism configuration, OpenAI-compatible API, and batching strategy all remain the same. You can also adjust context length on memory-constrained nodes with the --max-model-len parameter.

What This Means for the Ecosystem

The Transformers library has become the reference modeling library for the ML ecosystem over the past few years. As stated in the official blog post, Transformers is now positioned as a "model definition library": if a model is supported in Transformers, the goal is for it to be available across the entire ecosystem (vLLM, SGLang, MLX, llama.cpp).

This update is a concrete step toward that vision. Model authors can now:

1. Add their model to Transformers.
2. Get native-speed inference in vLLM with --model-impl transformers.
3. Skip writing an extra vLLM implementation.
4. Use the same code for training and evaluation.

This is also significant for teams doing enterprise LLM deployment. The time to bring new models into production drops dramatically. A model added to Transformers on a given day can run at native speed in vLLM that same day.

vLLM's Large Scale Serving Optimizations

This Transformers backend update is part of vLLM's broader performance journey. According to vLLM's large scale serving post, DeepSeek-style MoE models have reached 2.2k tok/s/H200 throughput. Optimizations like Wide-EP, Dual-batch Overlap (DBO), Expert Parallel Load Balancing (EPLB), and disaggregated serving are reducing inference costs.

The Transformers backend's compatibility with these optimizations means model authors can automatically benefit from them. Expert Parallelization support in MoE models shows that even large models like DeepSeek can be efficiently served with the Transformers backend.

Limitations and Caveats

• Linear attention models are not currently supported, but will be added soon. This may leave some research-oriented models out of scope.
• Custom Hub models (where code lives in a Hub repo) may not work if not written compliantly. These models need to follow Transformers compatibility rules.
• Performance gains depend on the model's architecture. Gains may be significant for some models and less noticeable for others.
• Automatic parallelization plans may not be derivable for all models, but the number of supported architectures is continuously growing.

This development is also important for small language models: the Transformers backend + vLLM combination can be used to optimize models running on edge devices.

Conclusion

The Transformers modeling backend reaching native speed in vLLM marks a milestone for the ML ecosystem. Model authors now get both Transformers and vLLM support with a single integration. The need for hand-written vLLM implementations decreases, and new models can reach production faster.

Techniques like torch.fx and AST manipulation that apply optimizations at runtime transform Transformers code to be compatible with vLLM's custom kernels. The Qwen3 benchmark results demonstrate that this approach works in practice. Achieving native performance across models ranging from 4B to 235B parameters is proof that this technology has matured.

If you're looking to improve LLM inference performance, update with uv pip install --upgrade vllm --torch-backend auto and add the --model-impl transformers flag. The results will speak for themselves.

If you have questions or experiences on this topic, feel free to share in the comments. Check out our blog to follow developments in AI and software development.

---

The model's working principles and benchmark data were compiled from the official Hugging Face blog and the vLLM blog.