Xenova/Qwen1.5-0.5B-Chat

Text generation · served with ONNX Runtime · ↓ 5.2K downloads a month · View on Hugging Face

These ONNX files are an export for running in a browser: GridPup serves them as tensors in and out, not as a chat. For chat, pick the same model in its original form (served with vLLM).

Weights: 1.9 GB
GPU memory needed: about 3.8 GB
Memory · disk: 8 GB · 30 GB
Licence: not stated

Versions

Smaller versions (8-bit, 4-bit) need less GPU memory and answer a little less precisely.

full precisiononnx/decoder_model_merged.onnx1.9 GB · 3.8 GB GPU 8-bitonnx/decoder_model_merged_quantized.onnx480 MB · 1.7 GB GPU full precisiononnx/model.onnx1.9 GB · 3.8 GB GPU 4-bit (bnb)onnx/model_bnb4.onnx810 MB · 2.2 GB GPU float16onnx/model_fp16.onnx940 MB · 2.4 GB GPU 8-bitonnx/model_int8.onnx480 MB · 1.7 GB GPU 4-bitonnx/model_q4.onnx830 MB · 2.2 GB GPU 4-bit, float16onnx/model_q4f16.onnx490 MB · 1.7 GB GPU 8-bitonnx/model_uint8.onnx480 MB · 1.7 GB GPU

Versions of Qwen1.5 0.5B

The same model packaged differently: a quantized version needs less GPU memory and runs on a cheaper machine.

Version	Runtime	Weights	GPU memory	Cheapest machine now
OriginalQwen/Qwen1.5-0.5B-Chat	vLLM	1.2 GB	4.5 GB	$0.10/h RTX 3060
Main (Xenova)Xenova/Qwen1.5-0.5B-Chat	ONNX Runtime	480 MB	1.7 GB	$0.08/h GTX 1660 SUPER

Hardware and deploy

Deploying ONNX models from the catalog is coming soon. Today, download the files from Hugging Face and upload them in Inference.

Loading…