How much GPU memory does Meta-Llama-Guard-2-8B need?

About 21.5 GB for conversations of up to 8,192 tokens: 16.06 GB of weights, the cache for one full-length conversation and the runtime's own memory (an estimate: its settings are behind its licence).

What is the cheapest GPU to run Meta-Llama-Guard-2-8B?

Right now 2x RTX 3060 at $0.18 an hour; RTX 5090 at $0.69 an hour is the best value for its speed. Prices are live offers from GPU clouds, checked every 15 minutes.

Can Meta-Llama-Guard-2-8B run on an RTX 4090?

Yes: it needs about 21.5 GB and an RTX 4090 has 24 GB.

Models / Meta-Llama-Guard-2

meta-llama/Meta-Llama-Guard-2-8B

Text generation · served with vLLM · ↓ 1.9K downloads a month · View on Hugging Face

meta-llama/Meta-Llama-Guard-2-8B needs about 21.5 GB of GPU memory for conversations of up to 8,192 tokens (an estimate: its settings are behind its licence). It runs on 2x RTX 3060 from $0.18/h right now; best value: RTX 5090 at $0.69/h.

The model's publisher asks users to accept its licence (llama3). GridPup cannot download this one yet: choose an ungated version below.

Weights: 16 GB
GPU memory needed: about 21.5 GB (estimate)
Conversation length: 8,192 tokens
Memory · disk: 16 GB · 58 GB
Architecture: LlamaForCausalLM
Licence: llama3

Hardware and deploy

The publisher has not let GridPup download this licence-gated model yet. An ungated version of it (above) can be deployed.

Questions

How much GPU memory does Meta-Llama-Guard-2-8B need?: About 21.5 GB for conversations of up to 8,192 tokens: 16.06 GB of weights, the cache for one full-length conversation and the runtime's own memory (an estimate: its settings are behind its licence).
What is the cheapest GPU to run Meta-Llama-Guard-2-8B?: Right now 2x RTX 3060 at $0.18 an hour; RTX 5090 at $0.69 an hour is the best value for its speed. Prices are live offers from GPU clouds, checked every 15 minutes.
Can Meta-Llama-Guard-2-8B run on an RTX 4090?: Yes: it needs about 21.5 GB and an RTX 4090 has 24 GB.

Loading…