> ## Documentation Index
> Fetch the complete documentation index at: https://wwwpoc.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# vLLM (Containerized)

> Learn how to run Granite models with vLLM in a container.

## Overview

In this guide, we'll use [vLLM](https://docs.vllm.ai/en/latest/) running inside a container to serve Granite models.

***

## Prerequisites

* A container runtime such as [Docker Desktop](https://www.docker.com/products/docker-desktop/) or [Podman](https://podman.io/)
* An NVIDIA GPU with drivers installed

***

## 1. Pull the vLLM container image

```sh theme={null}
docker pull vllm/vllm-openai:latest
```

*(For stability, you may pin a version tag, e.g., `vllm/vllm-openai:v0.18.0`. Granite models require `vllm` version 0.18.0 and above.)*

***

## 2. Run Granite in the container

Run the container with your Hugging Face cache mounted:

```sh theme={null}
docker run --runtime nvidia --gpus all     -v ~/.cache/huggingface:/root/.cache/huggingface     -p 8000:8000     vllm/vllm-openai:latest     --model ibm-granite/granite-4.0-h-small
```

<Note>
  You can pre-download the model into `~/.cache/huggingface` using:

  ```sh theme={null}
  hf download ibm-granite/granite-4.0-h-small
  ```

  If not pre-downloaded, the model will be fetched automatically when vLLM starts and cached in the mounted directory.
</Note>

***

## 3. Run a sample request

```sh theme={null}
curl -X POST http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
        "model": "ibm-granite/granite-4.0-h-small",
        "messages": [
          {"role": "user", "content": "How are you today?"}
        ]
      }'
```

## 4. Enabling tool calling and other extended capabilities

To run vLLM with the Granite 4.0 models and enabling capabilities such as tool calling, use the additional parameters `--tool-call-parser granite4` and `--enable-auto-tool-choice`. Refer to the vLLM documentation [here](https://docs.vllm.ai/en/stable/features/tool_calling/#ibm-granite) for more details on these and other parameters.

Now run the container with the added parameters:

```sh theme={null}
docker run --runtime nvidia --gpus all     -v ~/.cache/huggingface:/root/.cache/huggingface     -p 8000:8000     vllm/vllm-openai:latest     --model ibm-granite/granite-4.0-h-small --tool-call-parser granite4 --enable-auto-tool-choice
```

Once the container is up, you can start running requests using the OpenAI API. Refer to the documentation on OpenAI API [tool calling](/granite/docs/models/granite#tool-calling) for examples.

To run vLLM with the Granite 3 models and tool calling, use the additional parameters specified in the vLLM documentation [here](https://docs.vllm.ai/en/stable/features/tool_calling/#ibm-granite) as part of the `docker run` command share in Section 2.