How much GPU memory does c4ai-command-r-plus-4bit need?

About 77.7 GB for conversations of up to 8,192 tokens: 62.92 GB of weights, the cache for one full-length conversation and the runtime's own memory (an estimate: its settings are behind its licence).

What is the cheapest GPU to run c4ai-command-r-plus-4bit?

Right now 4x RTX 3090 at $1.04 an hour; 2x RTX PRO 5000 at $1.34 an hour is the best value for its speed. Prices are live offers from GPU clouds, checked every 15 minutes.

Can c4ai-command-r-plus-4bit run on an RTX 4090?

Not on one: it needs about 77.7 GB and an RTX 4090 has 24 GB. Use a bigger GPU, or a smaller (quantized) version of the model.

Models / c4ai-command-r-plus-4bit

CohereLabs/c4ai-command-r-plus-4bit

Text generation · served with vLLM · ↓ 173 downloads a month · View on Hugging Face

CohereLabs/c4ai-command-r-plus-4bit needs about 77.7 GB of GPU memory for conversations of up to 8,192 tokens (an estimate: its settings are behind its licence). It runs on 4x RTX 3090 from $1.04/h right now; best value: 2x RTX PRO 5000 at $1.34/h.

The model's publisher asks users to accept its licence (cc-by-nc-4.0). GridPup cannot download this one yet: choose an ungated version below.

Weights: 63 GB
GPU memory needed: about 77.7 GB (estimate)
Conversation length: 8,192 tokens
Memory · disk: 39.5 GB · 110 GB
Architecture: CohereForCausalLM
Licence: cc-by-nc-4.0

Hardware and deploy

The publisher has not let GridPup download this licence-gated model yet. An ungated version of it (above) can be deployed.

Questions

How much GPU memory does c4ai-command-r-plus-4bit need?: About 77.7 GB for conversations of up to 8,192 tokens: 62.92 GB of weights, the cache for one full-length conversation and the runtime's own memory (an estimate: its settings are behind its licence).
What is the cheapest GPU to run c4ai-command-r-plus-4bit?: Right now 4x RTX 3090 at $1.04 an hour; 2x RTX PRO 5000 at $1.34 an hour is the best value for its speed. Prices are live offers from GPU clouds, checked every 15 minutes.
Can c4ai-command-r-plus-4bit run on an RTX 4090?: Not on one: it needs about 77.7 GB and an RTX 4090 has 24 GB. Use a bigger GPU, or a smaller (quantized) version of the model.

Loading…