The Ultimate vLLM Setup Guide for 2025: 7 Key Steps

nvidia-smi

# Create a new directory for your project
mkdir vllm-project && cd vllm-project

# Create a virtual environment
python -m venv venv

# Activate it
source venv/bin/activate

pip install vllm

Step 3: Choose Your Language Model

vLLM can serve almost any decoder-based model from the Hugging Face Hub. The question is, which one should you choose?

Your choice depends on a balance of performance, quality, and your GPU's VRAM. A 70-billion parameter model is powerful but requires a ton of memory. A 7-billion parameter model is much lighter but might not have the same reasoning capabilities.

Quantization is Key for Efficiency

For most applications, a full-precision (FP16) model is overkill. Quantization is a technique that reduces the memory footprint of a model by using lower-precision numbers (like 8-bit or 4-bit integers) with minimal impact on performance. By 2025, this is standard practice.

Look for models with suffixes like AWQ or GPTQ. These are popular and effective quantization methods fully supported by vLLM.

For this guide, let's assume we're using a popular, efficient model like mistralai/Mistral-7B-Instruct-v0.2. For a quantized version, you might search the Hub for something like TheBloke/Mistral-7B-Instruct-v0.2-AWQ.

Step 4: Launch the API Server

This is where the magic happens. vLLM includes a built-in API server that mimics the OpenAI API, making it incredibly easy to integrate into existing applications.

To launch the server, run the following command, replacing the model name with your choice:

python -m vllm.entrypoints.openai.api_server \
    --model "mistralai/Mistral-7B-Instruct-v0.2"

vLLM will download the model from the Hugging Face Hub (if not already cached) and start the server. You should see output indicating the server is running, usually on http://localhost:8000.

Useful Server Arguments

• --model <model_name>: (Required) The Hugging Face model ID.
• --quantization <method>: Use awq or gptq if you're loading a quantized model. vLLM can often infer this, but it's good to be explicit.
• --tensor-parallel-size <N>: If you have multiple GPUs, set this to the number of GPUs you want to use (e.g., --tensor-parallel-size 2 for two GPUs). This splits the model across them.
• --host <ip_address>: Use --host 0.0.0.0 to make the server accessible from other machines on your network.
• --port <port_number>: Change the default port from 8000.

Step 5: Test Your API Endpoint

With the server running, let's make sure it works. Because vLLM uses an OpenAI-compatible API, you can use familiar tools.

Quick Test with cURL

Open a new terminal and use curl to send a request. This is the quickest way to verify the server is responding.

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{ 
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "prompt": "San Francisco is a",
    "max_tokens": 7,
    "temperature": 0
}'

You should get a JSON response back with the model's completion, something like: `"... a city in Northern California..."`.

Integrating into an Application with Python

For a more realistic use case, you can use the `openai` Python library. Just point it at your local server.

First, install the library: `pip install openai`

Then, run this Python script:

import openai

# Point to our local server
client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed" # vLLM doesn't require an API key by default
)

completion = client.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    prompt="The ultimate answer to life, the universe, and everything is",
    max_tokens=20,
    temperature=0.7
)

print(completion.choices[0].text)

Step 6: Tune for Peak Performance

Your server is running, but is it running optimally? The default settings are conservative. To unlock vLLM's full potential, you'll want to tune a few key parameters when launching the server.

• --gpu-memory-utilization <0.1-1.0>: This tells vLLM what fraction of the GPU's VRAM it's allowed to use. The default is 0.9 (90%). If this is the only process on the GPU, you can safely set it higher, like 0.95, to allow for larger batches. Don't set it to 1.0, as the OS and other processes need a little headroom.
• --max-model-len <length>: This sets the maximum context length the model can handle. A larger value allows for longer prompts but consumes more memory. Check your model's maximum supported length (e.g., 4096, 8192, or 32768 tokens) and set this accordingly, but be mindful of your VRAM.
• --max-num-seqs <number>: The maximum number of sequences (i.e., concurrent requests) to be processed in a single batch. Increasing this can improve throughput but also increases memory usage.

Example of a tuned launch command:

python -m vllm.entrypoints.openai.api_server \
    --model "mistralai/Mistral-7B-Instruct-v0.2" \
    --gpu-memory-utilization 0.95 \
    --max-model-len 8192

Step 7: Troubleshoot Common Issues

Even with a perfect guide, you might hit a snag. Here are solutions to the most common problems:

• Error: CUDA out of memory
  This is the classic one. It means your model and the current batch of requests won't fit in your GPU's VRAM.
  Solutions:
  1. Use a smaller model or a quantized (AWQ/GPTQ) version.
  2. Reduce --max-model-len or --max-num-seqs.
  3. If you have multiple GPUs, use --tensor-parallel-size to split the load.
• Error: Model files not found or compatibility error
  This can happen if the model name is wrong or if the model's configuration isn't fully supported.
  Solutions:
  1. Double-check the model ID on Hugging Face.
  2. Ensure you have git and git-lfs installed for downloading large files.
  3. Try a different, more popular model to confirm your setup is working.
• Performance is slow
  If inference feels sluggish, check your resource usage.
  Solutions:
  1. Run nvidia-smi in another terminal while the server is under load. Is GPU utilization near 100%? If not, the bottleneck might be your CPU or data input pipeline.
  2. Ensure you've set --gpu-memory-utilization appropriately to allow vLLM to use the hardware fully.

Conclusion

Congratulations! You've successfully set up, launched, and tested a high-performance LLM inference server with vLLM. You're now equipped to serve language models at speeds and throughput levels that were once the exclusive domain of large tech companies.

From here, the journey is one of experimentation. Try different models, tweak your performance parameters, and see how vLLM can accelerate your own AI-powered projects. The world of efficient LLM deployment is now at your fingertips.