How much GPU memory does ms-marco-MiniLM-L-4-v2 need?

About 1.1 GB: 0.08 GB of weights and the runtime's own memory.

What is the cheapest GPU to run ms-marco-MiniLM-L-4-v2?

Right now one GTX 1660 SUPER at $0.07 an hour; RTX 5090 at $0.62 an hour is the best value for its speed. Prices are live offers from GPU clouds, checked every 15 minutes.

Can ms-marco-MiniLM-L-4-v2 run on an RTX 4090?

Yes: it needs about 1.1 GB and an RTX 4090 has 24 GB.

Models / MiniLM-L12-H384-uncased / MiniLM-L12-H384-uncased

Xenova/ms-marco-MiniLM-L-4-v2

Classification · served with ONNX Runtime · ↓ 2.6K downloads a month · View on Hugging Face

Xenova/ms-marco-MiniLM-L-4-v2 needs about 1.1 GB of GPU memory. It runs on one GTX 1660 SUPER from $0.07/h right now; best value: RTX 5090 at $0.62/h.

Weights: 80 MB
GPU memory needed: about 1.1 GB
Memory · disk: 8 GB · 30 GB
Licence: not stated

Versions

Smaller versions (8-bit, 4-bit) need less GPU memory and answer a little less precisely.

full precisiononnx/model.onnx80 MB · 1.1 GB GPU 4-bit (bnb)onnx/model_bnb4.onnx50 MB · 1.1 GB GPU float16onnx/model_fp16.onnx40 MB · 1.1 GB GPU 8-bitonnx/model_int8.onnx20 MB · 1 GB GPU 4-bitonnx/model_q4.onnx50 MB · 1.1 GB GPU 4-bit, float16onnx/model_q4f16.onnx30 MB · 1 GB GPU 8-bitonnx/model_quantized.onnx20 MB · 1 GB GPU 8-bitonnx/model_uint8.onnx20 MB · 1 GB GPU

Versions of MiniLM-L12-H384-uncased

The same model packaged differently: a quantized version needs less GPU memory and runs on a cheaper machine.

Version	Runtime	Weights	GPU memory	Cheapest machine now
ms marco MiniLM L6 v2 (cross-encoder)cross-encoder/ms-marco-MiniLM-L6-v2	vLLM	50 MB	1.6 GB	$0.09/h RTX 3060
ms marco MiniLM L4 v2 (cross-encoder)cross-encoder/ms-marco-MiniLM-L4-v2	vLLM	40 MB	1.5 GB	$0.09/h RTX 3060
all MiniLM L12 v2 (sentence-transformers)sentence-transformers/all-MiniLM-L12-v2	vLLM	70 MB	1.6 GB	$0.09/h RTX 3060
ms marco MiniLM L12 v2 (cross-encoder)cross-encoder/ms-marco-MiniLM-L12-v2	vLLM	70 MB	1.6 GB	$0.09/h RTX 3060
ms marco MiniLM L 6 v2 (Xenova)Xenova/ms-marco-MiniLM-L-6-v2	ONNX Runtime	20 MB	1 GB	$0.08/h GTX 1660 SUPER
ms marco MiniLM L2 v2 (cross-encoder)cross-encoder/ms-marco-MiniLM-L2-v2

Hardware and deploy

Deploying ONNX models from the catalog is coming soon. Today, download the files from Hugging Face and upload them in Inference.

Questions

How much GPU memory does ms-marco-MiniLM-L-4-v2 need?: About 1.1 GB: 0.08 GB of weights and the runtime's own memory.
What is the cheapest GPU to run ms-marco-MiniLM-L-4-v2?: Right now one GTX 1660 SUPER at $0.07 an hour; RTX 5090 at $0.62 an hour is the best value for its speed. Prices are live offers from GPU clouds, checked every 15 minutes.
Can ms-marco-MiniLM-L-4-v2 run on an RTX 4090?: Yes: it needs about 1.1 GB and an RTX 4090 has 24 GB.

Loading…