OpenAI APIs - Vision

OpenAI APIs - Vision#

SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the OpenAI API Reference. This tutorial covers the vision APIs for vision language models.

SGLang supports vision language models such as Llama 3.2, LLaVA-OneVision, and QWen-VL2
- meta-llama/Llama-3.2-11B-Vision-Instruct
- lmms-lab/llava-onevision-qwen2-72b-ov-chat
- Qwen/Qwen2-VL-7B-Instruct

Launch A Server#

This code block is equivalent to executing

python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
  --port 30000 --chat-template llama_3_vision

in your terminal and wait for the server to be ready.

Remember to add --chat-template llama_3_vision to specify the vision chat template, otherwise the server only supports text. We need to specify --chat-template for vision language models because the chat template provided in Hugging Face tokenizer only supports text.

[1]:

from sglang.utils import (
    execute_shell_command,
    wait_for_server,
    terminate_process,
    print_highlight,
)

embedding_process = execute_shell_command(
    """
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
    --port=30000 --chat-template=llama_3_vision
"""
)

wait_for_server("http://localhost:30000")

[2024-12-04 19:25:08] server_args=ServerArgs(model_path='meta-llama/Llama-3.2-11B-Vision-Instruct', tokenizer_path='meta-llama/Llama-3.2-11B-Vision-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, context_length=None, device='cuda', served_model_name='meta-llama/Llama-3.2-11B-Vision-Instruct', chat_template='llama_3_vision', is_embedding=False, revision=None, host='127.0.0.1', port=30000, mem_fraction_static=0.88, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='lpm', schedule_conservativeness=1.0, cpu_offload_gb=0, tp_size=1, stream_interval=1, random_seed=619559929, constrained_json_whitespace_pattern=None, watchdog_timeout=300, download_dir=None, base_gpu_id=0, log_level='info', log_level_http=None, log_requests=False, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_pth='SGLang_storage', enable_cache_report=False, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, lora_paths=None, max_loras_per_batch=8, attention_backend='flashinfer', sampling_backend='flashinfer', grammar_backend='outlines', disable_radix_cache=False, disable_jump_forward=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, num_continuous_decode_steps=1, delete_ckpt_after_loading=False)
[2024-12-04 19:25:15] Use chat template for the OpenAI-compatible API server: llama_3_vision
[2024-12-04 19:25:25 TP0] Overlap scheduler is disabled for multimodal models.
[2024-12-04 19:25:26 TP0] Automatically turn off --chunked-prefill-size and adjust --mem-fraction-static for multimodal models.
[2024-12-04 19:25:26 TP0] Init torch distributed begin.
[2024-12-04 19:25:26 TP0] Load weight begin. avail mem=78.59 GB
[2024-12-04 19:25:26 TP0] lm_eval is not installed, GPTQ may not be usable
[2024-12-04 19:25:27 TP0] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:00<00:03,  1.19it/s]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:01<00:02,  1.13it/s]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:02<00:01,  1.08it/s]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:03<00:00,  1.06it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:04<00:00,  1.35it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:04<00:00,  1.22it/s]

[2024-12-04 19:25:31 TP0] Load weight end. type=MllamaForConditionalGeneration, dtype=torch.bfloat16, avail mem=58.43 GB
[2024-12-04 19:25:31 TP0] Memory pool end. avail mem=11.80 GB
[2024-12-04 19:25:31 TP0] Capture cuda graph begin. This can take up to several minutes.
[2024-12-04 19:25:39 TP0] Capture cuda graph end. Time elapsed: 8.24 s
[2024-12-04 19:25:41 TP0] max_total_num_tokens=298440, max_prefill_tokens=16384, max_running_requests=2049, context_len=131072
[2024-12-04 19:25:41] INFO:     Started server process [799665]
[2024-12-04 19:25:41] INFO:     Waiting for application startup.
[2024-12-04 19:25:41] INFO:     Application startup complete.
[2024-12-04 19:25:41] INFO:     Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
[2024-12-04 19:25:41] INFO:     127.0.0.1:58750 - "GET /v1/models HTTP/1.1" 200 OK
[2024-12-04 19:25:42] INFO:     127.0.0.1:58762 - "GET /get_model_info HTTP/1.1" 200 OK
[2024-12-04 19:25:42 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, cache hit rate: 0.00%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:25:42] INFO:     127.0.0.1:58778 - "POST /generate HTTP/1.1" 200 OK
[2024-12-04 19:25:42] The server is fired up and ready to roll!


NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.

Using cURL#

Once the server is up, you can send test requests using curl or requests.

[2]:

import subprocess

curl_command = """
curl -s http://localhost:30000/v1/chat/completions \
  -d '{
    "model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What’s in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
            }
          }
        ]
      }
    ],
    "max_tokens": 300
  }'
"""

response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)

[2024-12-04 19:25:49 TP0] Prefill batch. #new-seq: 1, #new-token: 6463, #cached-token: 0, cache hit rate: 0.00%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:25:50 TP0] Decode batch. #running-req: 1, #token: 6496, token usage: 0.02, gen throughput (token/s): 4.25, #queue-req: 0
[2024-12-04 19:25:50 TP0] Decode batch. #running-req: 1, #token: 6536, token usage: 0.02, gen throughput (token/s): 109.30, #queue-req: 0
[2024-12-04 19:25:50] INFO:     127.0.0.1:38146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{"id":"c0109cfdd5be49caba0024919be5ff65","object":"chat.completion","created":1733340350,"model":"meta-llama/Llama-3.2-11B-Vision-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The image shows a man standing behind a yellow taxi cab with an ironing board attached to the back of it, and an iron in his hand. He is wearing a yellow jacket and is standing on the street, with the taxi cab parked on the road. The man is ironing a piece of clothing that is draped over the back of the ironing board. In the background, there are other cars and buildings visible."},"logprobs":null,"finish_reason":"stop","matched_stop":128009}],"usage":{"prompt_tokens":6463,"total_tokens":6549,"completion_tokens":86,"prompt_tokens_details":null}}

Using Python Requests#

[3]:

import requests

url = "http://localhost:30000/v1/chat/completions"

data = {
    "model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
                    },
                },
            ],
        }
    ],
    "max_tokens": 300,
}

response = requests.post(url, json=data)
print_highlight(response.text)

[2024-12-04 19:25:51 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 6462, cache hit rate: 49.97%, token usage: 0.02, #running-req: 0, #queue-req: 0
[2024-12-04 19:25:51] INFO:     127.0.0.1:38160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{"id":"9cb0221de0324d37b070d5550d9986fc","object":"chat.completion","created":1733340351,"model":"meta-llama/Llama-3.2-11B-Vision-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The image shows a man ironing clothes on an ironing board that is mounted on the back of a yellow taxi cab."},"logprobs":null,"finish_reason":"stop","matched_stop":128009}],"usage":{"prompt_tokens":6463,"total_tokens":6489,"completion_tokens":26,"prompt_tokens_details":null}}

Using OpenAI Python Client#

[4]:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in this image?",
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
                    },
                },
            ],
        }
    ],
    max_tokens=300,
)

print_highlight(response.choices[0].message.content)

[2024-12-04 19:25:52 TP0] Prefill batch. #new-seq: 1, #new-token: 11, #cached-token: 6452, cache hit rate: 66.58%, token usage: 0.02, #running-req: 0, #queue-req: 0
[2024-12-04 19:25:52 TP0] Decode batch. #running-req: 1, #token: 6466, token usage: 0.02, gen throughput (token/s): 30.45, #queue-req: 0
[2024-12-04 19:25:52] INFO:     127.0.0.1:38166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
The image depicts a man ironing clothes on the back of a yellow taxi cab, which is parked on the street.

Multiple-Image Inputs#

The server also supports multiple images and interleaved text and images if the model supports it.

[5]:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true",
                    },
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png",
                    },
                },
                {
                    "type": "text",
                    "text": "I have two very different images. They are not related at all. "
                    "Please describe the first image in one sentence, and then describe the second image in another sentence.",
                },
            ],
        }
    ],
    temperature=0,
)

print_highlight(response.choices[0].message.content)

[2024-12-04 19:25:52 TP0] Prefill batch. #new-seq: 1, #new-token: 12895, #cached-token: 0, cache hit rate: 39.99%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:25:53 TP0] Decode batch. #running-req: 1, #token: 12914, token usage: 0.04, gen throughput (token/s): 31.36, #queue-req: 0
[2024-12-04 19:25:53 TP0] Decode batch. #running-req: 1, #token: 12954, token usage: 0.04, gen throughput (token/s): 108.73, #queue-req: 0
[2024-12-04 19:25:53] INFO:     127.0.0.1:56972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
The first image shows a man in a yellow shirt ironing a shirt on the back of a yellow taxi cab, with a small icon of a computer code snippet in the bottom left corner. The second image shows a large orange "S" and "G" and "L" on a white background.
[6]:

terminate_process(embedding_process)

Chat Template#

As mentioned before, if you do not specify a vision model’s --chat-template, the server uses Hugging Face’s default template, which only supports text.

We list popular vision models with their chat templates: