onnx-community/Qwen2.5-0.5B-Instruct

Text generation · served with ONNX Runtime · ↓ 45.3K downloads a month · View on Hugging Face

These ONNX files are an export for running in a browser: GridPup serves them as tensors in and out, not as a chat. For chat, pick the same model in its original form (served with vLLM).

Weights: 2.0 GB
GPU memory needed: about 4 GB
Memory · disk: 8 GB · 30 GB
Licence: not stated

Versions

Smaller versions (8-bit, 4-bit) need less GPU memory and answer a little less precisely.

full precisiononnx/model.onnx2.0 GB · 4 GB GPU 4-bit (bnb)onnx/model_bnb4.onnx760 MB · 2.1 GB GPU float16onnx/model_fp16.onnx1.0 GB · 2.5 GB GPU 8-bitonnx/model_int8.onnx510 MB · 1.8 GB GPU 4-bitonnx/model_q4.onnx790 MB · 2.2 GB GPU 4-bit, float16onnx/model_q4f16.onnx480 MB · 1.7 GB GPU 8-bitonnx/model_quantized.onnx510 MB · 1.8 GB GPU 8-bitonnx/model_uint8.onnx510 MB · 1.8 GB GPU

Versions of Qwen2.5 0.5B

The same model packaged differently: a quantized version needs less GPU memory and runs on a cheaper machine.

Version	Runtime	Weights	GPU memory	Cheapest machine now
OriginalQwen/Qwen2.5-0.5B	vLLM	990 MB	3.4 GB	$0.10/h RTX 3060
InstructQwen/Qwen2.5-0.5B-Instruct	vLLM	990 MB	3.4 GB	$0.10/h RTX 3060
Qwen2.5 Coder 0.5B InstructQwen/Qwen2.5-Coder-0.5B-Instruct	vLLM	990 MB	3.4 GB	$0.10/h RTX 3060
Qwen2.5 Coder 0.5BQwen/Qwen2.5-Coder-0.5B	vLLM	990 MB	3.4 GB	$0.10/h RTX 3060
Instruct AWQQwen/Qwen2.5-0.5B-Instruct-AWQ	vLLM	730 MB	3.1 GB	$0.10/h RTX 3060
turn detector (livekit)livekit/turn-detector	ONNX Runtime	170 MB	1.2 GB	$0.08/h GTX 1660 SUPER

Hardware and deploy

Deploying ONNX models from the catalog is coming soon. Today, download the files from Hugging Face and upload them in Inference.

Loading…