Quick Start

Launch a server

This code uses subprocess.Popen to start an SGLang server process, equivalent to executing

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 --host 0.0.0.0 --log-level warning

in your command line and wait for the server to be ready.

import subprocess
import time
import requests
import os

server_process = subprocess.Popen(
    [
        "python",
        "-m",
        "sglang.launch_server",
        "--model-path",
        "meta-llama/Meta-Llama-3.1-8B-Instruct",
        "--port",
        "30000",
        "--host",
        "0.0.0.0",
        "--log-level",
        "error",
    ],
    text=True,
    stdout=subprocess.DEVNULL,
    stderr=subprocess.DEVNULL,
)

while True:
    try:
        response = requests.get(
            "http://localhost:30000/v1/models",
            headers={"Authorization": "Bearer None"},
        )
        if response.status_code == 200:
            break
    except requests.exceptions.RequestException:
        time.sleep(1)

print("Server is ready. Proceeding with the next steps.")

Server is ready. Proceeding with the next steps.

Send a Request

Once the server is running, you can send test requests using curl.

!curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer None" \
  -d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is a LLM?"}]}'

{"id":"1449c9c20d4448299431a57facc68d7a","object":"chat.completion","created":1729816891,"model":"meta-llama/Meta-Llama-3.1-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"LLM stands for Large Language Model. It's a type of artificial intelligence (AI) designed to process and generate human-like language. LLMs are trained on vast amounts of text data, which enables them to learn patterns, relationships, and nuances of language.\n\nLarge Language Models are typically trained using a technique called deep learning, where multiple layers of artificial neural networks are used to analyze and understand the input data. This training process involves feeding the model massive amounts of text data, which it uses to learn and improve its language understanding and generation capabilities.\n\nSome key characteristics of LLMs include:\n\n1. **Language understanding**: LLMs can comprehend natural language, including its syntax, semantics, and context.\n2. **Language generation**: LLMs can generate text, including responses to user input, articles, stories, and more.\n3. **Contextual understanding**: LLMs can understand the context in which language is being used, including the topic, tone, and intent.\n4. **Self-supervised learning**: LLMs can learn from large datasets without explicit supervision or labeling.\n\nLLMs have a wide range of applications, including:\n\n1. **Virtual assistants**: LLMs power virtual assistants like Siri, Alexa, and Google Assistant.\n2. **Language translation**: LLMs can translate text from one language to another.\n3. **Text summarization**: LLMs can summarize long pieces of text into shorter, more digestible versions.\n4. **Content generation**: LLMs can generate content, such as news articles, product descriptions, and social media posts.\n5. **Chatbots**: LLMs can power chatbots that can have human-like conversations with users.\n\nThe Large Language Model I am, is a type of LLM that has been trained on a massive dataset of text and can answer a wide range of questions and engage in conversation."},"logprobs":null,"finish_reason":"stop","matched_stop":128009}],"usage":{"prompt_tokens":47,"total_tokens":426,"completion_tokens":379,"prompt_tokens_details":null}}

Using OpenAI Compatible API

SGLang supports OpenAI-compatible APIs. Here are Python examples:

import openai

# Always assign an api_key, even if not specified during server initialization.
# Setting an API key during server initialization is strongly recommended.

client = openai.Client(
    base_url="http://127.0.0.1:30000/v1", api_key="None"
)

# Chat completion example

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant"},
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)
print(response)

ChatCompletion(id='16757c3dd6e14a6e9bafd1122f84e4c5', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Here are 3 countries and their capitals:\n\n1. **Country:** Japan\n**Capital:** Tokyo\n\n2. **Country:** Australia\n**Capital:** Canberra\n\n3. **Country:** Brazil\n**Capital:** Brasília', refusal=None, role='assistant', function_call=None, tool_calls=None), matched_stop=128009)], created=1729816893, model='meta-llama/Meta-Llama-3.1-8B-Instruct', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=46, prompt_tokens=49, total_tokens=95, prompt_tokens_details=None))

import signal
import gc
import torch

def terminate_process(process):
    try:
        process.terminate()
        try:
            process.wait(timeout=5)
        except subprocess.TimeoutExpired:
            if os.name != 'nt':
                try:
                    pgid = os.getpgid(process.pid)
                    os.killpg(pgid, signal.SIGTERM)
                    time.sleep(1)
                    if process.poll() is None:
                        os.killpg(pgid, signal.SIGKILL)
                except ProcessLookupError:
                    pass
            else:
                process.kill()
            process.wait()
    except Exception as e:
        print(f"Warning: {e}")
    finally:
        gc.collect()
        if torch.cuda.is_available():
            torch.cuda.empty_cache()
            torch.cuda.ipc_collect()

terminate_process(server_process)
time.sleep(2)