Cluster

I recently ran into an interesting thermal issue while running two NVIDIA DGX Sparks as a small distributed inference cluster using llama.cpp. The two Sparks are directly connected over QSFP and use RDMA as the transport. With llama-server running on Spark1, and ggml-rpc-server running on Spark2, I’m using llama.cpp’s layer-split configuration with the DeepSeek-V4-Flash-0731-UD-Q4_K_XL model. The model is too large to run on a single Spark, thus the need to layer it over two Sparks. ...