How much GPU memory does AI21-Jamba-Large-1.5 need?

About 958.8 GB for conversations of up to 8,192 tokens: 797.11 GB of weights, the cache for one full-length conversation and the runtime's own memory (an estimate: its settings are behind its licence).

What is the cheapest GPU to run AI21-Jamba-Large-1.5?

Right now 8x Nvidia B200 at $82.72 an hour. Prices are live offers from GPU clouds, checked every 15 minutes.

Can AI21-Jamba-Large-1.5 run on an RTX 4090?

Not on one: it needs about 958.8 GB and an RTX 4090 has 24 GB. Use a bigger GPU, or a smaller (quantized) version of the model.

Models / AI21-Jamba-Large-1.5

ai21labs/AI21-Jamba-Large-1.5

Text generation · served with vLLM · ↓ 31 downloads a month · View on Hugging Face

ai21labs/AI21-Jamba-Large-1.5 needs about 958.8 GB of GPU memory for conversations of up to 8,192 tokens (an estimate: its settings are behind its licence). It runs on 8x Nvidia B200 from $82.72/h right now.

The model's publisher asks users to accept its licence (other). GridPup cannot download this one yet: choose an ungated version below.

Weights: 797 GB
GPU memory needed: about 958.8 GB (estimate)
Conversation length: 8,192 tokens
Memory · disk: 406.6 GB · 917 GB
Architecture: JambaForCausalLM
Licence: other

Hardware and deploy

The publisher has not let GridPup download this licence-gated model yet. An ungated version of it (above) can be deployed.

Questions

How much GPU memory does AI21-Jamba-Large-1.5 need?: About 958.8 GB for conversations of up to 8,192 tokens: 797.11 GB of weights, the cache for one full-length conversation and the runtime's own memory (an estimate: its settings are behind its licence).
What is the cheapest GPU to run AI21-Jamba-Large-1.5?: Right now 8x Nvidia B200 at $82.72 an hour. Prices are live offers from GPU clouds, checked every 15 minutes.
Can AI21-Jamba-Large-1.5 run on an RTX 4090?: Not on one: it needs about 958.8 GB and an RTX 4090 has 24 GB. Use a bigger GPU, or a smaller (quantized) version of the model.

Loading…