OpenAI APIs - Embedding#

SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the OpenAI API Reference.

This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the corresponding overview page

Launch A Server#

Launch the server in your terminal and wait for it to initialize. Remember to add --is-embedding to the command.

[1]:

from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

embedding_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
    --host 0.0.0.0 --is-embedding --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=embedding_process)

Skipping import of cpp extensions due to incompatible torch version. Please upgrade to torch >= 2.11.0 (found 2.9.1+cu130).

Skipping import of cpp extensions due to incompatible torch version. Please upgrade to torch >= 2.11.0 (found 2.9.1+cu130).
/actions-runner/_work/sglang/sglang/python/sglang/launch_server.py:54: UserWarning: 'python -m sglang.launch_server' is still supported, but 'sglang serve' is the recommended entrypoint.
  Example: sglang serve --model-path <model> [options]
  warnings.warn(
[2026-04-24 11:41:32] No platform detected. Using base SRTPlatform with defaults.
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-04-24 11:41:33] `torch_dtype` is deprecated! Use `dtype` instead!
`BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead.
[2026-04-24 11:41:34] `BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead.
Skipping import of cpp extensions due to incompatible torch version. Please upgrade to torch >= 2.11.0 (found 2.9.1+cu130).
Skipping import of cpp extensions due to incompatible torch version. Please upgrade to torch >= 2.11.0 (found 2.9.1+cu130).
No platform detected. Using base SRTPlatform with defaults.
No platform detected. Using base SRTPlatform with defaults.
`BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead.
`BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead.
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-04-24 11:41:42] `torch_dtype` is deprecated! Use `dtype` instead!
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
Multi-thread loading shards: 100% Completed | 2/2 [00:02<00:00,  1.17s/it]
2026-04-24 11:41:48,282 - CUTE_DSL - WARNING - [handle_import_error] - Unexpected error during package walk: cutlass.cute.experimental
[2026-04-24 11:41:48] Unexpected error during package walk: cutlass.cute.experimental
Compiling num tokens (num_tokens=4): 100%|██████████| 58/58 [00:04<00:00, 13.84it/s]
Capturing num tokens (num_tokens=4 avail_mem=131.45 GB): 100%|██████████| 58/58 [00:02<00:00, 27.53it/s]
/usr/local/lib/python3.10/dist-packages/fastapi/routing.py:120: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/
  response = await f(request)

NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
To reduce the log length, we set the log level to warning for the server, the default log level is info.
We are running those notebooks in a CI environment, so the throughput is not representative of the actual performance.

Using cURL#

[2]:

import subprocess, json

text = "Once upon a time"

curl_text = f"""curl -s http://localhost:{port}/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": "{text}"}}'"""

result = subprocess.check_output(curl_text, shell=True)

print(result)

text_embedding = json.loads(result)["data"][0]["embedding"]

print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")

b'{"data":[{"embedding":[-0.00023698806762695312,-0.0499267578125,-0.0032749176025390625,0.0110931396484375,-0.01406097412109375,0.016021728515625,-0.01444244384765625,0.005901336669921875,-0.022796630859375,0.0272979736328125,0.0015134811401367188,0.048370361328125,-0.001544952392578125,0.04522705078125,-0.01074981689453125,-0.00980377197265625,0.0230255126953125,0.0271453857421875,0.0090789794921875,0.01212310791015625,-0.0236358642578125,-0.00960540771484375,-0.039276123046875,-0.0252227783203125,0.0003082752227783203,0.0022430419921875,-0.010040283203125,-0.00238037109375,0.0251312255859375,0.00011104345321655273,-0.0236053466796875,-0.006168365478515625,-0.0087890625,0.053009033203125,0.00457000732421875,-0.024871826171875,-0.009765625,0.0307464599609375,-0.0050048828125,0.0185699462890625,0.0175018310546875,0.004169464111328125,-0.02264404296875,0.004085540771484375,0.03826904296875,0.007205963134765625,-0.0176849365234375,0.0282745361328125,-0.0002315044403076172,0.0209808349609375,-0.024505615234375,-0.012115478515625,-0.00527191162109375,-0.004970550537109375,0.029388427734375,-0.005535125732421875,0.01335906982421875,-0.00775146484375,0.0030117034912109375,-0.01418304443359375,0.0012836456298828125,0.0198211669921875,0.00023829936981201172,-0.00026726722717285156,-0.01143646240234375,0.004863739013671875,-0.017852783203125,-0.0141754150390625,0.00733184814453125,0.004207611083984375,0.0212554931640625,-0.005725860595703125,0.01666259765625,0.01519012451171875,-0.0137481689453125,-0.0011415481567382812,-0.01552581787109375,-0.006450653076171875,-0.00170135498046875,0.0252532958984375,0.0204010009765625,0.01084136962890625,0.00028061866760253906,-0.045989990234375,-0.00131988525390625,0.0066070556640625,0.0275115966796875,-0.01236724853515625,0.006290435791015625,0.0213775634765625,0.0168609619140625,0.00014412403106689453,0.1868896484375,0.0260162353515625,-0.033233642578125,0.014678955078125,-0.0222015380859375,-0.0224456787109375,-0.0173187255859375,0.00921630859375,0.02618408203125,-0.002044677734375,-0.00734710693359375,-0.0137481689453125,-0.0157623291015625,-0.0270233154296875,-0.369873046875,-0.0021152496337890625,0.0098724365234375,0.0028438568115234375,0.0282440185546875,-0.01221466064453125,-0.0262908935546875,-0.041656494140625,-0.0156402587890625,-0.0018367767333984375,0.0145263671875,-0.00496673583984375,-0.009857177734375,0.0079345703125,0.0052490234375,-0.02734375,-0.006988525390625,0.0005550384521484375,-0.0369873046875,0.09466552734375,0.021240234375,-0.01079559326171875,-0.04833984375,-0.0129241943359375,0.0199737548828125,0.00539398193359375,-0.021392822265625,0.0030498504638671875,0.0272369384765625,-0.03973388671875,-0.0017175674438476562,-0.011505126953125,0.005886077880859375,-0.00377655029296875,0.0020809173583984375,-0.005153656005859375,-0.010589599609375,0.01027679443359375,-0.028045654296875,-0.02825927734375,0.020721435546875,0.006595611572265625,0.0142974853515625,0.001888275146484375,-0.0003573894500732422,-0.01490020751953125,0.0096282958984375,0.00499725341796875,-0.00923919677734375,0.054107666015625,0.011932373046875,0.01080322265625,0.0233612060546875,0.026519775390625,-0.00591278076171875,0.0081024169921875,0.01108551025390625,-0.006183624267578125,0.00577545166015625,-0.0233154296875,0.0311126708984375,0.00079345703125,-0.035125732421875,0.0010929107666015625,0.003406524658203125,-0.0167999267578125,0.020477294921875,0.0093841552734375,0.001003265380859375,-0.0028171539306640625,-0.00955963134765625,0.0052337646484375,-0.01421356201171875,-0.041961669921875,0.0157012939453125,0.0110626220703125,0.006389617919921875,0.0138092041015625,0.007198333740234375,-0.0178985595703125,0.0018873214721679688,0.002788543701171875,-0.09197998046875,-0.006328582763671875,-0.0021114349365234375,0.006015777587890625,0.0253753662109375,-0.00522613525390625,0.0216064453125,-0.0001264810562133789,-0.01508331298828125,-0.008697509765625,0.0117950439453125,-0.0282440185546875,-0.00119781494140625,-0.00452423095703125,-0.00905609130859375,-0.0062103271484375,0.0286865234375,-0.004852294921875,-0.01293182373046875,-0.024139404296875,0.0085601806640625,-0.029052734375,-0.0035724639892578125,0.007282257080078125,-0.0172271728515625,-0.0120086669921875,0.004016876220703125,-0.01129150390625,-0.002307891845703125,-0.0213775634765625,-0.006717681884765625,0.022308349609375,-0.006900787353515625,-0.03802490234375,-0.0110321044921875,-0.002223968505859375,0.00872802734375,-0.00018703937530517578,-0.005023956298828125,0.0137176513671875,-0.0064239501953125,-0.0241241455078125,-0.004241943359375,-0.0028095245361328125,-0.00041174888610839844,-0.004650115966796875,-0.00035572052001953125,-0.005218505859375,0.023895263671875,0.00789642333984375,-0.01340484619140625,-0.019073486328125,0.01983642578125,0.00966644287109375,-0.0149688720703125,0.007511138916015625,-0.017547607421875,-0.006031036376953125,0.01313018798828125,0.0130767822265625,-0.01378631591796875,-0.007457733154296875,0.0157928466796875,-0.0209503173828125,-0.00859832763671875,0.005947113037109375,0.022308349609375,-0.01617431640625,0.015838623046875,-0.011505126953125,0.00377655029296875,0.0098114013671875,-0.0027751922607421875,0.028106689453125,-0.006252288818359375,-0.002162933349609375,-0.003627777099609375,0.0189208984375,0.0113372802734375,-0.01654052734375,-0.01457977294921875,-0.01287078857421875,-0.022430419921875,-0.0120086669921875,-0.0110015869140625,0.00357818603515625,0.0133056640625,-0.0026454925537109375,-0.0240631103515625,0.0161590576171875,0.034088134765625,-0.0073089599609375,0.0136566162109375,0.0067291259765625,0.023193359375,-0.0178375244140625,-0.0189208984375,-0.00806427001953125,-0.042449951171875,0.0138092041015625,0.0082855224609375,0.0241546630859375,-0.01056671142578125,-0.0031108856201171875,-0.01397705078125,0.00044155120849609375,0.026824951171875,-0.0144805908203125,0.01012420654296875,-0.010772705078125,-0.0219879150390625,-0.016326904296875,0.018768310546875,0.0031414031982421875,-0.00673675537109375,-0.005100250244140625,-0.01424407958984375,-0.01076507568359375,-0.0103759765625,0.0352783203125,-0.00666046142578125,-0.004039764404296875,0.0079193115234375,0.009429931640625,-0.00785064697265625,0.0116119384765625,-0.01934814453125,-0.0013589859008789062,0.013916015625,-0.004894256591796875,0.0011529922485351562,-0.0037250518798828125,0.01212310791015625,0.0091400146484375,0.038787841796875,0.01145172119140625,0.05914306640625,0.017852783203125,0.0243072509765625,0.001323699951171875,0.0136260986328125,-0.029388427734375,-0.028228759765625,-0.089599609375,0.01812744140625,0.0261688232421875,0.01157379150390625,0.0098419189453125,0.004772186279296875,0.04998779296875,0.0247802734375,-0.020538330078125,0.02581787109375,0.004611968994140625,-0.02471923828125,0.006549835205078125,-0.007381439208984375,-0.0239715576171875,-0.000530242919921875,0.0171966552734375,0.0298614501953125,0.0195465087890625,-0.009185791015625,-0.002658843994140625,-0.0011701583862304688,0.00209808349609375,-0.0184478759765625,0.02996826171875,-0.009002685546875,-0.00084686279296875,-0.0047149658203125,0.000965118408203125,-0.004886627197265625,0.024749755859375,0.00850677490234375,-0.01513671875,-0.0111541748046875,0.01308441162109375,0.0179290771484375,-0.03271484375,0.0238494873046875,-0.00106048583984375,-0.0175323486328125,-0.0007810592651367188,0.01898193359375,-0.015869140625,0.0007576942443847656,-0.00469207763671875,-0.00403594970703125,0.00870513916015625,-0.01108551025390625,-0.00901031494140625,-0.002288818359375,-0.007198333740234375,0.005931854248046875,0.01001739501953125,0.0204620361328125,0.0199737548828125,0.00836181640625,-0.02288818359375,-0.06915283203125,-0.01206207275390625,0.0380859375,-0.01073455810546875,0.0004107952117919922,-0.011077880859375,0.00045990943908691406,-0.0148468017578125,0.0035648345947265625,-0.030029296875,0.025299072265625,0.020477294921875,-0.0203399658203125,0.0014314651489257812,-0.03375244140625,-0.006534576416015625,-0.0025119781494140625,0.003147125244140625,0.00261688232421875,0.00601959228515625,0.02166748046875,-0.0082244873046875,0.00701141357421875,0.07745361328125,0.0020599365234375,-0.0030059814453125,-0.00142669677734375,-0.01212310791015625,-0.0242462158203125,0.0153350830078125,0.028961181640625,0.0016222000122070312,-0.01131439208984375,0.0016841888427734375,0.0162200927734375,0.004299163818359375,-0.01495361328125,0.008453369140625,0.0014495849609375,0.00027060508728027344,0.00670623779296875,-0.0239105224609375,-0.01305389404296875,0.01184844970703125,0.00884246826171875,-0.003070831298828125,-0.026580810546875,-0.0196990966796875,0.022308349609375,0.011260986328125,-0.01605224609375,-0.039306640625,0.033477783203125,0.0194549560546875,-0.0273895263671875,-0.006511688232421875,0.0283966064453125,-0.0131378173828125,-0.01763916015625,-0.0579833984375,0.006175994873046875,-0.0249176025390625,0.01953125,-0.007381439208984375,-0.0145263671875,-0.0019855499267578125,-0.00782012939453125,-0.0230865478515625,0.006381988525390625,-0.0230255126953125,-0.00115966796875,0.00609588623046875,0.0071258544921875,0.005908966064453125,-0.00899505615234375,0.0193328857421875,0.049713134765625,0.01540374755859375,-0.0009250640869140625,-0.028411865234375,0.012451171875,-0.019012451171875,0.004169464111328125,0.01079559326171875,-0.025543212890625,-0.2486572265625,-0.002849578857421875,-0.0216522216796875,0.01413726806640625,0.01727294921875,0.0305023193359375,-0.034698486328125,0.0184478759765625,-0.0191192626953125,-0.0167083740234375,-0.0208587646484375,-0.00829315185546875,0.037109375,0.0128936767578125,-0.0002282857894897461,0.0155029296875,0.031585693359375,0.01544189453125,-0.0009503364562988281,-0.005146026611328125,0.03399658203125,-0.00434112548828125,0.0105438232421875,0.0003485679626464844,-0.017059326171875,0.01065826416015625,-0.00905609130859375,0.010040283203125,0.01020050048828125,0.007381439208984375,0.0173797607421875,-0.020599365234375,0.0034942626953125,0.0006232261657714844,0.03228759765625,0.03253173828125,-0.00034999847412109375,-0.01160430908203125,0.007080078125,-0.01351165771484375,0.012603759765625,0.01078033447265625,0.042999267578125,0.0202178955078125,0.01021575927734375,0.004116058349609375,-0.006389617919921875,-0.003742218017578125,-0.0033473968505859375,-0.00890350341796875,0.0214996337890625,0.01136016845703125,0.00731658935546875,-0.038604736328125,-0.018768310546875,-0.016632080078125,-0.010894775390625,0.0090789794921875,0.038818359375,0.022674560546875,-0.007755279541015625,-0.007709503173828125,0.007549285888671875,-0.035247802734375,-0.0101776123046875,-0.0117950439453125,-0.0203094482421875,0.0014133453369140625,0.007720947265625,0.01178741455078125,-0.0018072128295898438,-0.04058837890625,0.004222869873046875,-0.00400543212890625,-0.003780364990234375,-0.0189666748046875,-0.00797271728515625,0.0260009765625,0.034515380859375,0.00818634033203125,-0.017364501953125,0.0014467239379882812,0.00174713134765625,-0.00811004638671875,-0.00833892822265625,-0.01261138916015625,-0.09515380859375,0.045318603515625,0.0020732879638671875,0.02252197265625,-0.0186004638671875,-0.01010894775390625,0.018280029296875,-0.0039825439453125,-0.01076507568359375,-0.0005035400390625,-0.01434326171875,-0.00614166259765625,-0.0154876708984375,0.0140380859375,-0.01029205322265625,-0.01336669921875,-0.01470184326171875,-0.00780487060546875,0.030670166015625,0.031005859375,0.00004506111145019531,-0.005687713623046875,-0.00891876220703125,0.00308990478515625,0.011505126953125,0.00453948974609375,0.0164337158203125,-0.00875091552734375,0.0084991455078125,-0.003292083740234375,0.0083465576171875,-0.01187896728515625,0.0115966796875,-0.013885498046875,-0.0107269287109375,0.0167083740234375,-0.130859375,-0.00995635986328125,-0.0121307373046875,0.0177459716796875,0.0137939453125,-0.01018524169921875,-0.00841522216796875,-0.0196990966796875,-0.020263671875,0.00646209716796875,-0.02557373046875,-0.0178375244140625,0.002685546875,-0.00815582275390625,0.00799560546875,0.00217437744140625,0.006580352783203125,-0.0270233154296875,-0.021453857421875,0.0019254684448242188,0.025665283203125,-0.015167236328125,-0.003749847412109375,-0.005657196044921875,-0.00004673004150390625,-0.0143585205078125,-0.004993438720703125,-0.0110626220703125,0.016845703125,0.00750732421875,0.01522064208984375,0.01226806640625,-0.0034770965576171875,-0.0135498046875,0.0158538818359375,0.0159149169921875,0.0217437744140625,0.02008056640625,-0.001277923583984375,-0.0148162841796875,0.0021724700927734375,-0.0343017578125,-0.0105438232421875,0.0101318359375,-0.01021575927734375,0.0310211181640625,0.015716552734375,-0.030029296875,-0.00042366981506347656,-0.02789306640625,-0.000858306884765625,0.01488494873046875,-0.004421234130859375,0.01959228515625,-0.003391265869140625,0.004177093505859375,0.008941650390625,-0.01483154296875,-0.028350830078125,-0.0286865234375,0.0136566162109375,0.00586700439453125,0.00804901123046875,0.006671905517578125,-0.026702880859375,0.01065826416015625,-0.00823211669921875,0.0027904510498046875,-0.04510498046875,0.014068603515625,-0.018524169921875,0.0150146484375,-0.01385498046875,-0.0083160400390625,-0.0211029052734375,-0.0038127899169921875,-0.01910400390625,-0.0255279541015625,0.006237030029296875,0.00962066650390625,0.016357421875,-0.01068878173828125,-0.00750732421875,-0.0260162353515625,-0.01373291015625,0.004032135009765625,-0.00789642333984375,-0.0038356781005859375,0.00873565673828125,-0.009124755859375,0.0100250244140625,0.01509857177734375,0.0015621185302734375,-0.00022864341735839844,0.000011861324310302734,-0.00982666015625,-0.0176849365234375,-0.016937255859375,0.0032787322998046875,-0.003147125244140625,0.0174407958984375,-0.01413726806640625,-0.0011615753173828125,-0.0157012939453125,0.0005326271057128906,-0.0109405517578125,-0.01010894775390625,-0.026092529296875,0.0208587646484375,-0.01418304443359375,0.009063720703125,0.002262115478515625,-0.0031375885009765625,-0.0157012939453125,-0.01323699951171875,0.05987548828125,-0.0631103515625,0.07958984375,-0.0032596588134765625,0.0021800994873046875,0.00960540771484375,-0.00882720947265625,0.00310516357421875,-0.003284454345703125,0.003997802734375,-0.01751708984375,-0.034637451171875,-0.018768310546875,0.0213623046875,0.01409149169921875,0.00659942626953125,0.0125274658203125,0.004970550537109375,0.010406494140625,0.0029239654541015625,-0.0153350830078125,-0.007762908935546875,-0.00894927978515625,0.0114898681640625,0.023284912109375,-0.0206756591796875,0.01038360595703125,0.073486328125,-0.006786346435546875,-0.0019168853759765625,0.01337432861328125,-0.015899658203125,0.04425048828125,-0.00970458984375,0.0723876953125,0.0093841552734375,-0.0224761962890625,-0.0154266357421875,0.0173797607421875,0.0328369140625,-0.0055389404296875,0.004970550537109375,0.0018558502197265625,-0.0302734375,-0.00203704833984375,-0.021270751953125,0.0144805908203125,-0.0117645263671875,-0.022857666015625,-0.01126861572265625,0.00539398193359375,-0.0018768310546875,-0.00614166259765625,0.0076446533203125,-0.0208282470703125,0.0002028942108154297,-0.0088348388671875,0.01129913330078125,0.0009479522705078125,0.016510009765625,-0.004695892333984375,0.0016317367553710938,-0.032440185546875,0.00637054443359375,-0.0477294921875,-0.0190582275390625,-0.01078033447265625,-0.0157318115234375,-0.01023101806640625,-0.037200927734375,-0.029327392578125,0.01226806640625,-0.0030841827392578125,0.0290679931640625,-0.009857177734375,0.02117919921875,-0.007476806640625,0.025238037109375,-0.005794525146484375,0.01103973388671875,0.0048370361328125,0.002460479736328125,-0.002048492431640625,0.0190582275390625,0.0011281967163085938,0.00582122802734375,0.0166473388671875,-0.0217437744140625,-0.067138671875,0.004695892333984375,-0.01131439208984375,-0.0079803466796875,0.174072265625,-0.005847930908203125,-0.012420654296875,0.0297393798828125,-0.001247406005859375,-0.031463623046875,0.01531982421875,0.00772857666015625,-0.034423828125,0.006053924560546875,0.0139007568359375,-0.00922393798828125,-0.01702880859375,0.034027099609375,0.060821533203125,-0.0018377304077148438,-0.009185791015625,0.037841796875,0.013885498046875,0.0025653839111328125,-0.0039520263671875,0.0289154052734375,0.01554107666015625,-0.0042572021484375,-0.001216888427734375,0.0095062255859375,0.0181121826171875,-0.0160064697265625,0.0186004638671875,0.00463104248046875,-0.026214599609375,-0.01071929931640625,0.0026092529296875,0.0135650634765625,-0.005237579345703125,0.01523590087890625,-0.0247955322265625,-0.0081939697265625,0.0010471343994140625,-0.01470184326171875,0.00861358642578125,0.0100555419921875,0.062744140625,0.016845703125,0.0093231201171875,0.016754150390625,-0.017059326171875,0.01031494140625,-0.00861358642578125,-0.0052947998046875,-0.0024433135986328125,0.01438140869140625,0.0081787109375,-0.004848480224609375,-0.01172637939453125,-0.01453399658203125,0.03704833984375,-0.00708770751953125,0.01236724853515625,-0.00627899169921875,-0.0097503662109375,0.006633758544921875,-0.025665283203125,0.0013065338134765625,-0.0111846923828125,0.0089874267578125,0.007694244384765625,0.0140533447265625,-0.0146636962890625,0.01074981689453125,0.033843994140625,0.00457000732421875,0.027618408203125,-0.0175933837890625,0.00396728515625,-0.0027904510498046875,-0.00952911376953125,-0.0236663818359375,-0.01165008544921875,-0.01116180419921875,0.003978729248046875,-0.0088348388671875,-0.01149749755859375,0.01490020751953125,0.08087158203125,-0.053497314453125,0.0026397705078125,0.0170440673828125,-0.007282257080078125,-0.00501251220703125,-0.03314208984375,0.0025959014892578125,0.00823974609375,0.019012451171875,-0.0177154541015625,-0.0229949951171875,-0.01220703125,0.00371551513671875,0.0023860931396484375,0.00742340087890625,-0.00567626953125,-0.003406524658203125,0.0272979736328125,0.00031685829162597656,0.021514892578125,-0.02313232421875,0.0008006095886230469,-0.0133209228515625,0.0160369873046875,0.00185394287109375,-0.0111541748046875,0.0040130615234375,-0.0016107559204101562,-0.0146331787109375,0.007129669189453125,0.00508880615234375,0.00731658935546875,0.0169830322265625,0.0291900634765625,-0.0310211181640625,-0.031341552734375,0.01256561279296875,0.01007843017578125,-0.02197265625,0.0135650634765625,0.0168304443359375,-0.07977294921875,-0.03662109375,-0.0200653076171875,-0.01073455810546875,-0.018341064453125,-0.003421783447265625,-0.0036907196044921875,-0.0073089599609375,0.005542755126953125,-0.02142333984375,-0.0016803741455078125,-0.0192413330078125,0.09442138671875,-0.005420684814453125,0.0084075927734375,-0.264404296875,-0.0184783935546875,-0.00800323486328125,-0.00739288330078125,-0.02520751953125,0.027313232421875,0.007785797119140625,-0.0037631988525390625,0.0160980224609375,0.0311126708984375,-0.01247406005859375,-0.0130767822265625,0.00925445556640625,0.06854248046875,0.01593017578125,0.040435791015625,-0.0243988037109375,0.02899169921875,0.01427459716796875,-0.00786590576171875,-0.0012903213500976562,-0.035888671875,0.0015411376953125,0.0005879402160644531,-0.0015897750854492188,0.01152801513671875,0.0146331787109375,-0.00914764404296875,-0.0196533203125,0.023193359375,-0.001132965087890625,-0.0003414154052734375,0.005847930908203125,-0.0078582763671875,-0.00962066650390625,0.0132904052734375,0.031982421875,-0.01457977294921875,0.0118865966796875,-0.004974365234375,-0.025299072265625,0.0008087158203125,0.01490020751953125,0.0159759521484375,0.0102081298828125,-0.00818634033203125,-0.0018520355224609375,-0.00579833984375,0.00852203369140625,0.0109100341796875,-0.021881103515625,-0.004535675048828125,-0.0038547515869140625,-0.00276947021484375,0.01360321044921875,-0.00392913818359375,-0.0021228790283203125,-0.010650634765625,-0.031646728515625,-0.00646209716796875,-0.0058135986328125,-0.0017976760864257812,-0.028106689453125,-0.0028743743896484375,0.0095062255859375,0.0030307769775390625,-0.0126190185546875,-0.0177001953125,-0.01070404052734375,-0.0406494140625,-0.00374603271484375,-0.007518768310546875,0.01558685302734375,-0.005580902099609375,-0.01361083984375,0.0172271728515625,0.007350921630859375,-0.0296630859375,0.0015306472778320312,-0.0216217041015625,-0.0291290283203125,-0.00521087646484375,-0.00940704345703125,0.0040130615234375,-0.030242919921875,0.005878448486328125,0.014678955078125,0.005451202392578125,-0.011505126953125,-0.0174560546875,-0.0010309219360351562,0.01457977294921875,0.0141448974609375,-0.016204833984375,-0.0013628005981445312,-0.0008020401000976562,0.0081634521484375,0.01413726806640625,-0.0064849853515625,0.01259613037109375,0.0156402587890625,-0.00396728515625,0.0009984970092773438,0.03643798828125,0.018951416015625,0.01446533203125,0.00223541259765625,-0.036468505859375,-0.0159912109375,-0.0038089752197265625,-0.0032958984375,-0.00922393798828125,-0.0125274658203125,-0.00836944580078125,-0.0963134765625,0.06121826171875,0.006824493408203125,0.022247314453125,0.009552001953125,-0.0196380615234375,-0.0036106109619140625,-0.021026611328125,0.056854248046875,-0.0024662017822265625,0.0148162841796875,-0.05084228515625,-0.0010051727294921875,-0.006526947021484375,-0.0186004638671875,0.01143646240234375,0.001983642578125,-0.002349853515625,0.00812530517578125,-0.007045745849609375,0.00960540771484375,0.0007042884826660156,0.01806640625,-0.01308441162109375,0.0026912689208984375,0.0030517578125,0.0088043212890625,-0.03094482421875,0.02166748046875,0.0044403076171875,-0.0241851806640625,-0.0133514404296875,0.0362548828125,-0.0626220703125,0.0294647216796875,0.01456451416015625,0.006931304931640625,-0.0151519775390625,-0.005428314208984375,0.006267547607421875,-0.00725555419921875,-0.01279449462890625,-0.0017242431640625,0.01171875,0.0025196075439453125,0.010986328125,-0.0247344970703125,0.005573272705078125,-0.01285552978515625,-0.00827789306640625,-0.0166015625,-0.010101318359375,0.024932861328125,0.01557159423828125,-0.01441192626953125,-0.1650390625,0.0226593017578125,0.034454345703125,-0.025970458984375,-0.006038665771484375,0.0167388916015625,0.01117706298828125,-0.0064544677734375,-0.00847625732421875,0.00756072998046875,0.01491546630859375,-0.015960693359375,-0.002414703369140625,0.028106689453125,0.00103759765625,0.04815673828125,-0.011474609375,-0.030364990234375,-0.03472900390625,0.038818359375,0.003932952880859375,-0.0019464492797851562,-0.01503753662109375,-0.00554656982421875,0.012603759765625,0.018768310546875,0.0301361083984375,-0.016876220703125,0.004077911376953125,0.026641845703125,-0.01134490966796875,-0.0052337646484375,0.01035308837890625,-0.001132965087890625,-0.0634765625,0.01219940185546875,-0.01082611083984375,-0.014373779296875,-0.0098419189453125,0.0220184326171875,0.00591278076171875,0.010498046875,-0.0158233642578125,-0.006877899169921875,-0.00853729248046875,-0.04852294921875,0.0111846923828125,0.00812530517578125,0.008331298828125,0.1513671875,0.008270263671875,0.02191162109375,0.0017766952514648438,-0.005462646484375,-0.004100799560546875,-0.011749267578125,0.022705078125,0.00457000732421875,0.029052734375,-0.0301666259765625,-0.002948760986328125,-0.00019371509552001953,0.0255889892578125,0.0214996337890625,0.01328277587890625,0.006603240966796875,0.004261016845703125,-0.0143585205078125,0.0185394287109375,-0.0131072998046875,-0.00005882978439331055,0.00902557373046875,-0.0149993896484375,-0.006122589111328125,0.0174407958984375,-0.01971435546875,-0.004688262939453125,-0.00717926025390625,-0.026214599609375,0.00463104248046875,0.006000518798828125,0.006946563720703125,-0.0174713134765625,-0.0125885009765625,0.03216552734375,-0.01678466796875,-0.0168304443359375,-0.01025390625,0.016845703125,-0.032806396484375,0.0035343170166015625,0.009185791015625,-0.0123291015625,-0.0009679794311523438,-0.0167999267578125,0.03387451171875,-0.0029850006103515625,0.040618896484375,-0.0125579833984375,0.00621795654296875,-0.031494140625,-0.003925323486328125,-0.036224365234375,-0.06927490234375,0.0184326171875,0.002849578857421875,-0.01274871826171875,-0.015838623046875,0.04144287109375,0.00254058837890625,-0.0333251953125,0.0013456344604492188,-0.0003228187561035156,-0.00513458251953125,-0.017852783203125,0.00043129920959472656,-0.02081298828125,0.007450103759765625,-0.0007009506225585938,0.00037598609924316406,0.040771484375,-0.0005626678466796875,0.0091552734375,-0.006778717041015625,0.001708984375,-0.0129852294921875,-0.003078460693359375,-0.020904541015625,-0.01116943359375,-0.0091400146484375,-0.00946044921875,-0.0068359375,-0.029571533203125,-0.0066986083984375,-0.006542205810546875,0.00865936279296875,0.05731201171875,0.01174163818359375,-0.03656005859375,0.03570556640625,0.003170013427734375,-0.01348876953125,-0.0020351409912109375,-0.018707275390625,0.00841522216796875,-0.01390838623046875,-0.0023021697998046875,0.000957489013671875,0.026458740234375,-0.01076507568359375,-0.0128326416015625,0.0001080632209777832,0.015777587890625,-0.00012117624282836914,0.0136871337890625,0.00301361083984375,-0.01523590087890625,0.018280029296875,0.0000775456428527832,0.039459228515625,-0.0182952880859375,0.0023250579833984375,0.015594482421875,0.0148468017578125,-0.01153564453125,-0.0301666259765625,-0.040252685546875,-0.01483154296875,0.0017271041870117188,0.0007548332214355469,0.0215301513671875,0.01140594482421875,0.020660400390625,0.0295562744140625,0.00690460205078125,-0.0196990966796875,0.0216217041015625,-0.0008726119995117188,0.001377105712890625,-0.01064300537109375,0.03204345703125,-0.0014400482177734375,0.0146331787109375,-0.008697509765625,-0.004177093505859375,0.0059814453125,0.0009531974792480469,-0.0166778564453125,-0.0172882080078125,0.0009756088256835938,-0.0247344970703125,0.0131378173828125,-0.00543975830078125,-0.0028324127197265625,-0.0044097900390625,0.00286102294921875,0.01456451416015625,0.0013189315795898438,0.00873565673828125,0.0034637451171875,0.028717041015625,-0.007793426513671875,0.01033782958984375,0.01535797119140625,-0.00009137392044067383,-0.0017375946044921875,-0.01099395751953125,-0.01013946533203125,-0.01284027099609375,0.00467681884765625,0.0144805908203125,0.01141357421875,0.0140838623046875,-0.001857757568359375,-0.0225067138671875,-0.005645751953125,0.00618743896484375,-0.01445770263671875,-0.01067352294921875,0.0208587646484375,0.0103302001953125,0.0200042724609375,-0.01177978515625,-0.017913818359375,0.0015964508056640625,0.0084991455078125,0.03289794921875,0.00015842914581298828,0.02716064453125,-0.01219940185546875,-0.025146484375,-0.004695892333984375,0.01053619384765625,-0.0130615234375,-0.011383056640625,-0.0030651092529296875,-0.001583099365234375,0.0119781494140625,0.0208282470703125,0.01189422607421875,0.06439208984375,-0.01010894775390625,-0.003509521484375,0.007415771484375,-0.0153045654296875,-0.0294647216796875,-0.0006756782531738281,-0.0218658447265625,-0.005035400390625,0.0012941360473632812,-0.0239715576171875,-0.01471710205078125,-0.00750732421875,-0.00994873046875,-0.01453399658203125,0.0467529296875,-0.0172882080078125,0.0015439987182617188,0.015899658203125,-0.0080108642578125,-0.0200347900390625,-0.006626129150390625,-0.0093994140625,0.0023746490478515625,-0.01885986328125,0.0285491943359375,-0.0190277099609375,0.00927734375,-0.001949310302734375,0.0113677978515625,-0.03448486328125,-0.0033893585205078125,0.002231597900390625,-0.005496978759765625,0.0006618499755859375,-0.0207061767578125,0.0360107421875,-0.01078033447265625,-0.00015413761138916016,-0.009735107421875,0.01012420654296875,0.006519317626953125,-0.012115478515625,-0.010284423828125,-0.00394439697265625,0.0374755859375,-0.0203704833984375,0.01025390625,-0.03253173828125,0.017578125,0.021240234375,0.0197296142578125,0.0300445556640625,0.0103302001953125,-0.018157958984375,-0.0033283233642578125,-0.01287078857421875,0.00627899169921875,0.01131439208984375,0.00359344482421875,-0.003204345703125,0.00734710693359375,-0.0149688720703125,0.1773681640625,-0.023162841796875,0.0207061767578125,-0.0123748779296875,-0.0037841796875,0.00933837890625,-0.0022869110107421875,-0.019805908203125,0.001857757568359375,-0.007694244384765625,0.0146636962890625,-0.01055908203125,0.01519012451171875,-0.01505279541015625,-0.012481689453125,0.006633758544921875,0.0261993408203125,0.01154327392578125,-0.0154876708984375,-0.0208282470703125,-0.00414276123046875,-0.0122222900390625,-0.00893402099609375,0.016632080078125,0.01374053955078125,-0.0027484893798828125,-0.018951416015625,0.0292205810546875,0.007053375244140625,-0.0003833770751953125,-0.011932373046875,-0.0159149169921875,-0.0086669921875,-0.0196685791015625,0.0037631988525390625,-0.0201873779296875,0.00887298583984375,-0.01300048828125,-0.0311431884765625,-0.0162200927734375,0.01377105712890625,0.01256561279296875,0.01326751708984375,0.0182342529296875,-0.0073089599609375,0.00616455078125,-0.0152587890625,0.01325225830078125,-0.0009694099426269531,0.0013513565063476562,-0.0284576416015625,0.00545501708984375,0.0010480880737304688,0.0182342529296875,-0.00472259521484375,0.0007138252258300781,0.01007843017578125,-0.005626678466796875,0.0178070068359375,0.0180816650390625,0.0217742919921875,-0.0214691162109375,-0.0279541015625,0.003997802734375,0.013671875,0.00589752197265625,-0.00608062744140625,-0.019561767578125,-0.01348114013671875,0.0156707763671875,0.03424072265625,-0.009185791015625,0.01535797119140625,0.0056304931640625,-0.0021495819091796875,0.0035991668701171875,0.0120086669921875,-0.002899169921875,0.005664825439453125,-0.00658416748046875,-0.0289306640625,-0.01525115966796875,0.01488494873046875,-0.036376953125,0.0032806396484375,-0.0041351318359375,0.0222320556640625,0.02325439453125,0.0011548995971679688,-0.0025463104248046875,-0.021514892578125,-0.0178070068359375,0.01235198974609375,-0.003963470458984375,0.01035308837890625,-0.012237548828125,-0.008392333984375,-0.00453948974609375,0.00005334615707397461,-0.00377655029296875,-0.003448486328125,-0.0156097412109375,-0.000980377197265625,0.0148468017578125,-0.01303863525390625,-0.001201629638671875,0.00991058349609375,-0.005008697509765625,-0.015869140625,-0.0139312744140625,-0.00853729248046875,0.0143585205078125,0.005443572998046875,-0.0033664703369140625,0.0010242462158203125,0.016845703125,-0.005817413330078125,0.05645751953125,0.00856781005859375,0.00872039794921875,0.016143798828125],"index":0,"object":"embedding"}],"model":"Alibaba-NLP/gte-Qwen2-1.5B-instruct","object":"list","usage":{"prompt_tokens":4,"total_tokens":4,"completion_tokens":0,"prompt_tokens_details":null,"reasoning_tokens":0}}'

Text embedding (first 10): [-0.00023698806762695312, -0.0499267578125, -0.0032749176025390625, 0.0110931396484375, -0.01406097412109375, 0.016021728515625, -0.01444244384765625, 0.005901336669921875, -0.022796630859375, 0.0272979736328125]

Using Python Requests#

[3]:

import requests

text = "Once upon a time"

response = requests.post(
    f"http://localhost:{port}/v1/embeddings",
    json={"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text},
)

text_embedding = response.json()["data"][0]["embedding"]

print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")

Using OpenAI Python Client#

[4]:

import openai

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

# Text embedding example
response = client.embeddings.create(
    model="Alibaba-NLP/gte-Qwen2-1.5B-instruct",
    input=text,
)

embedding = response.data[0].embedding[:10]
print_highlight(f"Text embedding (first 10): {embedding}")

Using Input IDs#

SGLang also supports input_ids as input to get the embedding.

[5]:

import json
import os
from transformers import AutoTokenizer

os.environ["TOKENIZERS_PARALLELISM"] = "false"

tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct")
input_ids = tokenizer.encode(text)

curl_ids = f"""curl -s http://localhost:{port}/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": {json.dumps(input_ids)}}}'"""

input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))["data"][
    0
]["embedding"]

print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")

Input IDs embedding (first 10): [-0.00023698806762695312, -0.0499267578125, -0.0032749176025390625, 0.0110931396484375, -0.01406097412109375, 0.016021728515625, -0.01444244384765625, 0.005901336669921875, -0.022796630859375, 0.0272979736328125]

[6]:

terminate_process(embedding_process)

OpenAI APIs - Embedding

Contents