OpenAI APIs - Completions#

SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the OpenAI API Reference.

This tutorial covers the following popular APIs:

chat/completions
completions
batches

Check out other tutorials to learn about vision APIs for vision-language models and embedding APIs for embedding models.

Launch A Server#

This code block is equivalent to executing

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 --host 0.0.0.0

in your terminal and wait for the server to be ready.

[1]:

from sglang.utils import (
    execute_shell_command,
    wait_for_server,
    terminate_process,
    print_highlight,
)

server_process = execute_shell_command(
    "python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 30000 --host 0.0.0.0"
)

wait_for_server("http://localhost:30000")

[2024-12-04 19:22:34] server_args=ServerArgs(model_path='meta-llama/Meta-Llama-3.1-8B-Instruct', tokenizer_path='meta-llama/Meta-Llama-3.1-8B-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, context_length=None, device='cuda', served_model_name='meta-llama/Meta-Llama-3.1-8B-Instruct', chat_template=None, is_embedding=False, revision=None, host='0.0.0.0', port=30000, mem_fraction_static=0.88, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='lpm', schedule_conservativeness=1.0, cpu_offload_gb=0, tp_size=1, stream_interval=1, random_seed=541118400, constrained_json_whitespace_pattern=None, watchdog_timeout=300, download_dir=None, base_gpu_id=0, log_level='info', log_level_http=None, log_requests=False, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_pth='SGLang_storage', enable_cache_report=False, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, lora_paths=None, max_loras_per_batch=8, attention_backend='flashinfer', sampling_backend='flashinfer', grammar_backend='outlines', disable_radix_cache=False, disable_jump_forward=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, num_continuous_decode_steps=1, delete_ckpt_after_loading=False)
[2024-12-04 19:22:50 TP0] Init torch distributed begin.
[2024-12-04 19:22:50 TP0] Load weight begin. avail mem=78.59 GB
[2024-12-04 19:22:50 TP0] lm_eval is not installed, GPTQ may not be usable
[2024-12-04 19:22:51 TP0] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:02,  1.25it/s]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.14it/s]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.09it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.45it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.32it/s]

[2024-12-04 19:22:54 TP0] Load weight end. type=LlamaForCausalLM, dtype=torch.bfloat16, avail mem=63.50 GB
[2024-12-04 19:22:54 TP0] Memory pool end. avail mem=8.37 GB
[2024-12-04 19:22:54 TP0] Capture cuda graph begin. This can take up to several minutes.
[2024-12-04 19:23:01 TP0] Capture cuda graph end. Time elapsed: 7.12 s
[2024-12-04 19:23:02 TP0] max_total_num_tokens=442913, max_prefill_tokens=16384, max_running_requests=2049, context_len=131072
[2024-12-04 19:23:02] INFO:     Started server process [797819]
[2024-12-04 19:23:02] INFO:     Waiting for application startup.
[2024-12-04 19:23:02] INFO:     Application startup complete.
[2024-12-04 19:23:02] INFO:     Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit)
[2024-12-04 19:23:02] INFO:     127.0.0.1:49860 - "GET /v1/models HTTP/1.1" 200 OK
[2024-12-04 19:23:03] INFO:     127.0.0.1:49868 - "GET /get_model_info HTTP/1.1" 200 OK
[2024-12-04 19:23:03 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, cache hit rate: 0.00%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:23:03] INFO:     127.0.0.1:49880 - "POST /generate HTTP/1.1" 200 OK
[2024-12-04 19:23:03] The server is fired up and ready to roll!

NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.

Chat Completions#

Usage#

The server fully implements the OpenAI API. It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available. You can also specify a custom chat template with --chat-template when launching the server.

[2]:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print_highlight(f"Response: {response}")

[2024-12-04 19:23:08 TP0] Prefill batch. #new-seq: 1, #new-token: 42, #cached-token: 1, cache hit rate: 2.00%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:23:08 TP0] Decode batch. #running-req: 1, #token: 76, token usage: 0.00, gen throughput (token/s): 6.49, #queue-req: 0
[2024-12-04 19:23:08] INFO:     127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Response: ChatCompletion(id='edc09f77c8ce4db8acedd2829601a25c', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Here are 3 countries and their capitals:\n\n1. Country: Japan\n Capital: Tokyo\n\n2. Country: Australia\n Capital: Canberra\n\n3. Country: Brazil\n Capital: Brasília', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None), matched_stop=128009)], created=1733340188, model='meta-llama/Meta-Llama-3.1-8B-Instruct', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=43, prompt_tokens=43, total_tokens=86, completion_tokens_details=None, prompt_tokens_details=None))

Parameters#

The chat completions API accepts OpenAI Chat Completions API’s parameters. Refer to OpenAI Chat Completions API for more details.

Here is an example of a detailed chat completion request:

[3]:

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {
            "role": "system",
            "content": "You are a knowledgeable historian who provides concise responses.",
        },
        {"role": "user", "content": "Tell me about ancient Rome"},
        {
            "role": "assistant",
            "content": "Ancient Rome was a civilization centered in Italy.",
        },
        {"role": "user", "content": "What were their major achievements?"},
    ],
    temperature=0.3,  # Lower temperature for more focused responses
    max_tokens=128,  # Reasonable length for a concise response
    top_p=0.95,  # Slightly higher for better fluency
    presence_penalty=0.2,  # Mild penalty to avoid repetition
    frequency_penalty=0.2,  # Mild penalty for more natural language
    n=1,  # Single response is usually more stable
    seed=42,  # Keep for reproducibility
)

print_highlight(response.choices[0].message.content)

[2024-12-04 19:23:08 TP0] Prefill batch. #new-seq: 1, #new-token: 51, #cached-token: 25, cache hit rate: 20.63%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:23:08 TP0] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2024-12-04 19:23:08 TP0] Decode batch. #running-req: 1, #token: 106, token usage: 0.00, gen throughput (token/s): 121.25, #queue-req: 0
[2024-12-04 19:23:09 TP0] Decode batch. #running-req: 1, #token: 146, token usage: 0.00, gen throughput (token/s): 143.07, #queue-req: 0
[2024-12-04 19:23:09 TP0] Decode batch. #running-req: 1, #token: 186, token usage: 0.00, gen throughput (token/s): 141.84, #queue-req: 0
[2024-12-04 19:23:09] INFO:     127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Ancient Rome's major achievements include:

1. **Law and Governance**: The Twelve Tables (450 BCE) and the Julian Laws (50 BCE) established a foundation for modern law, while the Roman Senate and Assemblies developed a system of governance.
2. **Engineering and Architecture**: Romans built impressive structures like the Colosseum (80 CE), Pantheon (126 CE), and aqueducts, showcasing their engineering prowess.
3. **Infrastructure**: They developed a network of roads (over 250,000 miles), bridges, and canals, facilitating trade and communication.
4. **Military Conquests**: Rome expanded its territories

Streaming mode is also supported.

[4]:

stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Say this is a test"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

[2024-12-04 19:23:09] INFO:     127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2024-12-04 19:23:09 TP0] Prefill batch. #new-seq: 1, #new-token: 10, #cached-token: 30, cache hit rate: 33.73%, token usage: 0.00, #running-req: 0, #queue-req: 0
This is only a test.

Completions#

Usage#

Completions API is similar to Chat Completions API, but without the messages parameter or chat templates.

[5]:

response = client.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    prompt="List 3 countries and their capitals.",
    temperature=0,
    max_tokens=64,
    n=1,
    stop=None,
)

print_highlight(f"Response: {response}")

[2024-12-04 19:23:09 TP0] Prefill batch. #new-seq: 1, #new-token: 8, #cached-token: 1, cache hit rate: 32.57%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:23:09 TP0] Decode batch. #running-req: 1, #token: 24, token usage: 0.00, gen throughput (token/s): 128.30, #queue-req: 0
[2024-12-04 19:23:10 TP0] Decode batch. #running-req: 1, #token: 64, token usage: 0.00, gen throughput (token/s): 148.01, #queue-req: 0
[2024-12-04 19:23:10] INFO:     127.0.0.1:49894 - "POST /v1/completions HTTP/1.1" 200 OK

Response: Completion(id='764e0d7a597e4a768a24b6365d92ffab', choices=[CompletionChoice(finish_reason='length', index=0, logprobs=None, text=' 1. 2. 3.\n1. United States - Washington D.C. 2. Japan - Tokyo 3. Australia - Canberra\nList 3 countries and their capitals. 1. 2. 3.\n1. China - Beijing 2. Brazil - Bras', matched_stop=None)], created=1733340190, model='meta-llama/Meta-Llama-3.1-8B-Instruct', object='text_completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=64, prompt_tokens=9, total_tokens=73, completion_tokens_details=None, prompt_tokens_details=None))

Parameters#

The completions API accepts OpenAI Completions API’s parameters. Refer to OpenAI Completions API for more details.

Here is an example of a detailed completions request:

[6]:

response = client.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    prompt="Write a short story about a space explorer.",
    temperature=0.7,  # Moderate temperature for creative writing
    max_tokens=150,  # Longer response for a story
    top_p=0.9,  # Balanced diversity in word choice
    stop=["\n\n", "THE END"],  # Multiple stop sequences
    presence_penalty=0.3,  # Encourage novel elements
    frequency_penalty=0.3,  # Reduce repetitive phrases
    n=1,  # Generate one completion
    seed=123,  # For reproducible results
)

print_highlight(f"Response: {response}")

[2024-12-04 19:23:10 TP0] Prefill batch. #new-seq: 1, #new-token: 9, #cached-token: 1, cache hit rate: 31.35%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:23:10 TP0] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2024-12-04 19:23:10 TP0] Decode batch. #running-req: 1, #token: 41, token usage: 0.00, gen throughput (token/s): 137.63, #queue-req: 0
[2024-12-04 19:23:10] INFO:     127.0.0.1:49894 - "POST /v1/completions HTTP/1.1" 200 OK

Response: Completion(id='db8b0b857770461f86f7546160dadd21', choices=[CompletionChoice(finish_reason='stop', index=0, logprobs=None, text=' (approx. 250-300 words)\nAs the stars whizzed by like diamonds on velvet, Captain Orion gazed out at the endless expanse of space. Her ship, the Celestial Quest, had been traveling for months, and she was the first human to ever set foot on the distant planet of Xylophia-IV.', matched_stop='\n\n')], created=1733340190, model='meta-llama/Meta-Llama-3.1-8B-Instruct', object='text_completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=70, prompt_tokens=10, total_tokens=80, completion_tokens_details=None, prompt_tokens_details=None))

Structured decoding (JSON, Regex)#

You can specify a JSON schema or a regular expression to constrain the model output. The model output will be guaranteed to follow the given constraints.

JSON#

[7]:

import json

json_schema = json.dumps(
    {
        "type": "object",
        "properties": {
            "name": {"type": "string", "pattern": "^[\\w]+$"},
            "population": {"type": "integer"},
        },
        "required": ["name", "population"],
    }
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Give me the information of the capital of France in the JSON format.",
        },
    ],
    temperature=0,
    max_tokens=128,
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "foo", "schema": json.loads(json_schema)},
    },
)

print_highlight(response.choices[0].message.content)

[2024-12-04 19:23:10 TP0] Prefill batch. #new-seq: 1, #new-token: 19, #cached-token: 30, cache hit rate: 37.61%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:23:10] INFO:     127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK

{"name": "Paris", "population": 2147000}

Regular expression#

[8]:

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "What is the capital of France?"},
    ],
    temperature=0,
    max_tokens=128,
    extra_body={"regex": "(Paris|London)"},
)

print_highlight(response.choices[0].message.content)

[2024-12-04 19:23:11 TP0] Prefill batch. #new-seq: 1, #new-token: 12, #cached-token: 30, cache hit rate: 42.75%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:23:11] INFO:     127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Paris

Batches#

Batches API for chat completions and completions are also supported. You can upload your requests in jsonl files, create a batch job, and retrieve the results when the batch job is completed (which takes longer but costs less).

The batches APIs are:

batches
batches/{batch_id}/cancel
batches/{batch_id}

Here is an example of a batch job for chat completions, completions are similar.

[9]:

import json
import time
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="None")

requests = [
    {
        "custom_id": "request-1",
        "method": "POST",
        "url": "/chat/completions",
        "body": {
            "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
            "messages": [
                {"role": "user", "content": "Tell me a joke about programming"}
            ],
            "max_tokens": 50,
        },
    },
    {
        "custom_id": "request-2",
        "method": "POST",
        "url": "/chat/completions",
        "body": {
            "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
            "messages": [{"role": "user", "content": "What is Python?"}],
            "max_tokens": 50,
        },
    },
]

input_file_path = "batch_requests.jsonl"

with open(input_file_path, "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

with open(input_file_path, "rb") as f:
    file_response = client.files.create(file=f, purpose="batch")

batch_response = client.batches.create(
    input_file_id=file_response.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

print_highlight(f"Batch job created with ID: {batch_response.id}")

[2024-12-04 19:23:11] INFO:     127.0.0.1:49936 - "POST /v1/files HTTP/1.1" 200 OK
[2024-12-04 19:23:11] INFO:     127.0.0.1:49936 - "POST /v1/batches HTTP/1.1" 200 OK

Batch job created with ID: batch_9c2aae59-5fd2-41b0-adbd-b9f327ab0f7d

[2024-12-04 19:23:11 TP0] Prefill batch. #new-seq: 2, #new-token: 18, #cached-token: 62, cache hit rate: 50.56%, token usage: 0.00, #running-req: 0, #queue-req: 0

[10]:

while batch_response.status not in ["completed", "failed", "cancelled"]:
    time.sleep(3)
    print(f"Batch job status: {batch_response.status}...trying again in 3 seconds...")
    batch_response = client.batches.retrieve(batch_response.id)

if batch_response.status == "completed":
    print("Batch job completed successfully!")
    print(f"Request counts: {batch_response.request_counts}")

    result_file_id = batch_response.output_file_id
    file_response = client.files.content(result_file_id)
    result_content = file_response.read().decode("utf-8")

    results = [
        json.loads(line) for line in result_content.split("\n") if line.strip() != ""
    ]

    for result in results:
        print_highlight(f"Request {result['custom_id']}:")
        print_highlight(f"Response: {result['response']}")

    print_highlight("Cleaning up files...")
    # Only delete the result file ID since file_response is just content
    client.files.delete(result_file_id)
else:
    print_highlight(f"Batch job failed with status: {batch_response.status}")
    if hasattr(batch_response, "errors"):
        print_highlight(f"Errors: {batch_response.errors}")

[2024-12-04 19:23:11 TP0] Decode batch. #running-req: 1, #token: 63, token usage: 0.00, gen throughput (token/s): 69.26, #queue-req: 0
Batch job status: validating...trying again in 3 seconds...
[2024-12-04 19:23:14] INFO:     127.0.0.1:49936 - "GET /v1/batches/batch_9c2aae59-5fd2-41b0-adbd-b9f327ab0f7d HTTP/1.1" 200 OK
Batch job completed successfully!
Request counts: BatchRequestCounts(completed=2, failed=0, total=2)
[2024-12-04 19:23:14] INFO:     127.0.0.1:49936 - "GET /v1/files/backend_result_file-2abb4285-d82f-41c6-9e01-e2f38b0693d9/content HTTP/1.1" 200 OK

Request request-1:

Response: {'status_code': 200, 'request_id': 'request-1', 'body': {'id': 'request-1', 'object': 'chat.completion', 'created': 1733340191, 'model': 'meta-llama/Meta-Llama-3.1-8B-Instruct', 'choices': {'index': 0, 'message': {'role': 'assistant', 'content': 'Why do programmers prefer dark mode?\n\nBecause light attracts bugs.'}, 'logprobs': None, 'finish_reason': 'stop', 'matched_stop': 128009}, 'usage': {'prompt_tokens': 41, 'completion_tokens': 13, 'total_tokens': 54}, 'system_fingerprint': None}}

Request request-2:

Response: {'status_code': 200, 'request_id': 'request-2', 'body': {'id': 'request-2', 'object': 'chat.completion', 'created': 1733340191, 'model': 'meta-llama/Meta-Llama-3.1-8B-Instruct', 'choices': {'index': 0, 'message': {'role': 'assistant', 'content': '**What is Python?**\n\nPython is a high-level, interpreted programming language that is widely used for various purposes such as web development, scientific computing, data analysis, artificial intelligence, and more. It was created in the late 1980s by'}, 'logprobs': None, 'finish_reason': 'length', 'matched_stop': None}, 'usage': {'prompt_tokens': 39, 'completion_tokens': 50, 'total_tokens': 89}, 'system_fingerprint': None}}

Cleaning up files...

[2024-12-04 19:23:14] INFO:     127.0.0.1:49936 - "DELETE /v1/files/backend_result_file-2abb4285-d82f-41c6-9e01-e2f38b0693d9 HTTP/1.1" 200 OK

It takes a while to complete the batch job. You can use these two APIs to retrieve the batch job status or cancel the batch job.

batches/{batch_id}: Retrieve the batch job status.
batches/{batch_id}/cancel: Cancel the batch job.

Here is an example to check the batch job status.

[11]:

import json
import time
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="None")

requests = []
for i in range(100):
    requests.append(
        {
            "custom_id": f"request-{i}",
            "method": "POST",
            "url": "/chat/completions",
            "body": {
                "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
                "messages": [
                    {
                        "role": "system",
                        "content": f"{i}: You are a helpful AI assistant",
                    },
                    {
                        "role": "user",
                        "content": "Write a detailed story about topic. Make it very long.",
                    },
                ],
                "max_tokens": 500,
            },
        }
    )

input_file_path = "batch_requests.jsonl"
with open(input_file_path, "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

with open(input_file_path, "rb") as f:
    uploaded_file = client.files.create(file=f, purpose="batch")

batch_job = client.batches.create(
    input_file_id=uploaded_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

print_highlight(f"Created batch job with ID: {batch_job.id}")
print_highlight(f"Initial status: {batch_job.status}")

time.sleep(10)

max_checks = 5
for i in range(max_checks):
    batch_details = client.batches.retrieve(batch_id=batch_job.id)

    print_highlight(
        f"Batch job details (check {i+1} / {max_checks}) // ID: {batch_details.id} // Status: {batch_details.status} // Created at: {batch_details.created_at} // Input file ID: {batch_details.input_file_id} // Output file ID: {batch_details.output_file_id}"
    )
    print_highlight(
        f"<strong>Request counts: Total: {batch_details.request_counts.total} // Completed: {batch_details.request_counts.completed} // Failed: {batch_details.request_counts.failed}</strong>"
    )

    time.sleep(3)

[2024-12-04 19:23:14] INFO:     127.0.0.1:45300 - "POST /v1/files HTTP/1.1" 200 OK
[2024-12-04 19:23:14] INFO:     127.0.0.1:45300 - "POST /v1/batches HTTP/1.1" 200 OK

Created batch job with ID: batch_06f43602-6b36-4f44-9feb-e244bbff3667

Initial status: validating

[2024-12-04 19:23:14 TP0] Prefill batch. #new-seq: 8, #new-token: 240, #cached-token: 200, cache hit rate: 47.74%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2024-12-04 19:23:14 TP0] Prefill batch. #new-seq: 92, #new-token: 2760, #cached-token: 2300, cache hit rate: 45.77%, token usage: 0.00, #running-req: 8, #queue-req: 0
[2024-12-04 19:23:14 TP0] Decode batch. #running-req: 100, #token: 4425, token usage: 0.01, gen throughput (token/s): 414.92, #queue-req: 0
[2024-12-04 19:23:14 TP0] Decode batch. #running-req: 100, #token: 8425, token usage: 0.02, gen throughput (token/s): 11889.61, #queue-req: 0
[2024-12-04 19:23:15 TP0] Decode batch. #running-req: 100, #token: 12425, token usage: 0.03, gen throughput (token/s): 11631.67, #queue-req: 0
[2024-12-04 19:23:15 TP0] Decode batch. #running-req: 100, #token: 16425, token usage: 0.04, gen throughput (token/s): 11361.20, #queue-req: 0
[2024-12-04 19:23:15 TP0] Decode batch. #running-req: 100, #token: 20425, token usage: 0.05, gen throughput (token/s): 11110.18, #queue-req: 0
[2024-12-04 19:23:16 TP0] Decode batch. #running-req: 100, #token: 24425, token usage: 0.06, gen throughput (token/s): 10854.44, #queue-req: 0
[2024-12-04 19:23:16 TP0] Decode batch. #running-req: 100, #token: 28425, token usage: 0.06, gen throughput (token/s): 10604.87, #queue-req: 0
[2024-12-04 19:23:17 TP0] Decode batch. #running-req: 100, #token: 32425, token usage: 0.07, gen throughput (token/s): 10366.97, #queue-req: 0
[2024-12-04 19:23:17 TP0] Decode batch. #running-req: 100, #token: 36425, token usage: 0.08, gen throughput (token/s): 10158.90, #queue-req: 0
[2024-12-04 19:23:17 TP0] Decode batch. #running-req: 100, #token: 40425, token usage: 0.09, gen throughput (token/s): 9940.30, #queue-req: 0
[2024-12-04 19:23:18 TP0] Decode batch. #running-req: 100, #token: 44425, token usage: 0.10, gen throughput (token/s): 9732.34, #queue-req: 0
[2024-12-04 19:23:18 TP0] Decode batch. #running-req: 100, #token: 48425, token usage: 0.11, gen throughput (token/s): 9544.98, #queue-req: 0
[2024-12-04 19:23:19 TP0] Decode batch. #running-req: 100, #token: 52425, token usage: 0.12, gen throughput (token/s): 9309.81, #queue-req: 0
[2024-12-04 19:23:24] INFO:     127.0.0.1:56356 - "GET /v1/batches/batch_06f43602-6b36-4f44-9feb-e244bbff3667 HTTP/1.1" 200 OK

Batch job details (check 1 / 5) // ID: batch_06f43602-6b36-4f44-9feb-e244bbff3667 // Status: completed // Created at: 1733340194 // Input file ID: backend_input_file-0715ade6-36fb-466d-998b-b8b03206f213 // Output file ID: backend_result_file-094bd83a-5af0-470a-a23a-637f87f053f1