Distributed Gemma 31B on two computers
I have two 16 GB GPUs at home, but in different computers: an RTX 4080 in the desktop and an RTX 5060 Ti in the home server. Gemma 4 31B in a decent 4-bit quant weighs 17 GB and doesn’t fit into either of them. You could take iq3, but then quality suffers, and you still can’t get a big context.
But llama.cpp has RPC: on one machine you run rpc-server, which simply serves up its GPU over the network, and on the other - a regular llama-server, which sees the remote card as just another device and spreads the model’s layers across them. So that’s what I tried.
How it works
- The desktop with the 4080 is the coordinator. It runs
llama-server, and the model can be taken straight from the LM Studio folder. - The server with the 5060 Ti is the worker. It runs
rpc-server24/7, and when idle it takes up 180 MB of VRAM. - Both are in docker, and the image has to be built by hand: the official llama.cpp images are built without RPC. And the image has to be byte-for-byte identical on both machines, so it’s built once and carried over with
docker save | docker load.
The model is gemma-4-31B-it-qat-UD-Q4_K_XL from unsloth. It’s QAT, meaning the model was fine-tuned with quantization in mind, and 4 bits cost it almost nothing. The weights are split roughly in half, the remote card goes first and the local one last - it holds the output layer, so the logits don’t have to travel over the network.
What came out of it
26 tokens per second on short requests. 64K context, and 4 parallel requests still fit on top of that: 23 tok/s each, about 50 in total.
The network barely gets in the way. Everything runs into GPU memory: every token means reading all 17 GB of weights, and the 5060 Ti’s memory is one and a half times slower than the 4080’s. Plus the cards work in turns, first one computes its layers, then the other. The theoretical ceiling from memory bandwidth is about 32 tok/s, so 26 is perfectly fair.
And then came the surprises.
MTP doesn’t work. On a single GPU it gave me a 2x speedup. Here the speed doesn’t grow, and on a long context it actually drops by half: 7 tok/s versus 14. Every step with MTP costs almost as much as three regular tokens, and the longer the context, the worse it gets. Looks like something in the implementation is shipping extra stuff over RPC. I went through the flags, nothing helped, so for now I turned it off.
KV cache in f16 is faster than in q8. Out of habit I set q8_0: half the memory, and therefore, I thought, faster too. But at 10K context f16 gives 23.5 tok/s versus 20. Apparently unpacking q8 is more expensive than reading twice as much. Though with 128K context f16 no longer fits, so I went with 64K and f16.
Another nice little thing: llama-server has --sleep-idle-seconds. When idle it unloads the model from both cards and loads it back on its own on the first request. The first reply after sleep arrives in about 45 seconds, but in exchange the VRAM isn’t occupied when it isn’t needed.
Wrapping up
- Two GPUs in different computers do add up into one through llama.cpp RPC, and a 31B model in 4 bits runs on them at a decent speed.
- Parallel requests are almost free: four at once go at almost the same speed as one.
- MTP over RPC unfortunately gives no speedup.
- A quantized KV cache saves memory, but can slow down generation. Worth measuring instead of setting q8 out of habit.