With a standard InfiniBand rail-optimized cluster like GreenNode, running a 1T parameter model across 2 H100 nodes (8 GPUs each) is more straightforward than most teams expect. This guide walks through the exact steps – scripts included.

The trickiest part is getting NCCL right – it's the communication layer that ties GPUs across nodes together. Get it wrong and the run hangs silently or crashes with cryptic timeout errors. The recipes below are validated on GreenNode's production clusters, so the NCCL configuration is already solved.

Why Kimi-K2.7-Code?

We chose Kimi-K2.7-Code for a few concrete reasons:

Native fp8 support. Most 1T-class models require a separate quantization step before serving. Kimi-K2.7-Code ships with built-in fp8 weights, so vLLM loads it directly – no quantization pass, no extra tooling.
MoE architecture fits multi-node well. Its Mixture-of-Experts design means only a subset of parameters are active per token, which maps efficiently to tensor + pipeline parallelism.
MLA (Multi-head Latent Attention). Compared to standard MHA, MLA compresses the KV cache significantly – 576 elements per layer instead of the full head dimension – which is why we can fit ~90 concurrent 32K-token sequences in just 96.6 GiB of KV cache.
Code + reasoning focus. Built-in --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 support is production-ready out of the box for agentic coding pipelines.

Memory sizing: why 16×H100 is the minimum

At fp8 precision, the weights alone require ~1,017 GiB. With 16×80GB = 1,280 GiB total VRAM, after accounting for activation memory and framework overhead, you're left with about 96 GiB for KV cache. That's tight but workable.

MEMORY BUDGET — Kimi-K2.7-Code (MoE/MLA) on 16x H100 80GB
GPUs                               : 16 x 80 GiB = 1,280.0 GiB
Usable (util 92% – 4 GiB/gpu act.): 1,113.6 GiB
Weights                            : 1,040 B params @ fp8 (x1.05) = 1,017.0 GiB
--> KV cache pool                  : 96.6 GiB
Attention                          : MLA latent (512+64=576 elems/layer), 61 layers
KV dtype                           : fp8
KV cost per token                  : 34.3 KiB
Total KV tokens                    : 2,951,927  (sum across all concurrent seqs)
At max-model-len=32,768:
  per full sequence : 1.1 GiB
  max concurrent    : ~90 full-length sequences

Disk space: The full model checkpoint is approximately 1.9 TB. Make sure /data has at least 2 TB of free space before starting the download.

Environment setup

Both nodes must have exactly the same versions installed:

CUDA: 13.3
libnccl2: 2.30.7-1+cuda13.3
vLLM: 0.23.0
vllm-flash-attn: 2.6.2
nvidia-nccl-cu13: 2.30.4

Version mismatch kills the run. NCCL is extremely sensitive to version differences between nodes. Verify with dpkg -l | grep nccl on both nodes before proceeding.

Download the model

Use huggingface-cli inside a tmux session – the download is large and will take several hours:

tmux new -s model-download
hf download --local-dir /data/kimi moonshotai/Kimi-K2.7-Code

Set the critical NCCL environment variable

Before launching anything, identify the InfiniBand or high-speed network interface name on your nodes (e.g., ib0, eth0, bond0) and export it:

export NCCL_SOCKET_IFNAME=ib0   # replace with your actual interface name
export NODE1_IP=<ip-of-first-node>

This tells NCCL which network interface to use for inter-node communication. Getting this wrong is the single most common reason for hangs – NCCL falls back to a slow or broken interface and stalls silently.

Option 1: vLLM native multi-node (recommended)

Based on the official vLLM recipe for Kimi-K2.7-Code. No additional cluster manager needed.

Key flags explained:

-tp 8 – Tensor parallelism across 8 GPUs per node. Must equal the number of GPUs per node.
-pp 2 – Pipeline parallelism across 2 nodes. Must equal the number of nodes.
--enable-expert-parallel – Distributes MoE expert weights across GPUs instead of replicating them. Critical for memory efficiency.
--tool-call-parser kimi_k2 – Activates Kimi's custom tool-call output parser.
--reasoning-parser kimi_k2 – Enables extraction of the model's chain-of-thought from the output stream.
--headless (node 2 only) – Runs as a worker without starting its own API server.

Node 1:

MODEL=/data/kimi/huggingface/hub/models--moonshotai--Kimi-K2.7-Code/snapshots/74797c9c62378b951a1f6fcf5c4631024e9b8bef
vllm serve $MODEL \
  --served-model-name kimi-k2 \
  -tp 8 -pp 2 \
  --trust-remote-code \
  --tool-call-parser kimi_k2 \
  --reasoning-parser kimi_k2 \
  --enable-auto-tool-choice \
  --enable-expert-parallel \
  --mm-encoder-tp-mode data \
  --master-addr $NODE1_IP \
  --nnodes 2 \
  --node-rank 0

Node 2:

vllm serve $MODEL \
  --served-model-name kimi-k2 \
  -tp 8 -pp 2 \
  --trust-remote-code \
  --tool-call-parser kimi_k2 \
  --reasoning-parser kimi_k2 \
  --enable-auto-tool-choice \
  --enable-expert-parallel \
  --mm-encoder-tp-mode data \
  --master-addr $NODE1_IP \
  --nnodes 2 \
  --node-rank 1 \
  --headless

After loading (~8–12 minutes), the API server is ready at http://$NODE1_IP:8000/v1. Verify with:

curl http://$NODE1_IP:8000/v1/models

Option 2: Ray cluster

Use this if you're running shared infrastructure or plan to serve multiple models. Ray gives you health checks, dynamic scaling, and easier restart recovery.

Step 1 – Start the Ray head node (node 1):

pip install "ray[default]"
ray start --head \
  --node-ip-address=$NODE1_IP \
  --port=6379 \
  --dashboard-host=0.0.0.0 \
  --dashboard-port=8265 \
  --num-gpus=8

Step 2 – Join node 2 as a worker:

ray start \
  --address=$NODE1_IP:6379 \
  --node-ip-address=$NODE2_IP \
  --num-gpus=8

Step 3 – Verify cluster health (should show 2 nodes, 16 GPUs total, no failures):

ray status

Step 4 – Launch vLLM via Ray:

python -m vllm.entrypoints.openai.api_server \
  --model $MODEL \
  --served-model-name kimi-k2 \
  -tp 16 \
  --trust-remote-code \
  --tool-call-parser kimi_k2 \
  --reasoning-parser kimi_k2 \
  --enable-auto-tool-choice \
  --enable-expert-parallel \
  --mm-encoder-tp-mode data

When using Ray, switch from -tp 8 -pp 2 to -tp 16 – Ray handles cross-node GPU coordination transparently.

Optimization: skip the vision encoder

Kimi-K2.7-Code includes MoonViT, a vision encoder for multimodal inputs. If your use case is text and code only, skip loading MoonViT entirely with --language-model-only. This frees VRAM directly into the KV cache pool.

vllm serve $MODEL \
  ... \
  --language-model-only

Default (with MoonViT): ~96.6 GiB KV cache, ~90 concurrent 32K sequences
With --language-model-only: ~130–140 GiB KV cache (est.), ~120–130 concurrent sequences (est.)

For a shared API endpoint with multiple users, this is an easy ~30–40% throughput gain at no cost.

Troubleshooting

Run hangs at startup – no progress after "initializing distributed environment"

Cause: NCCL is using the wrong network interface. Fix: Verify NCCL_SOCKET_IFNAME is set correctly on both nodes. Enable debug logging to see which interface NCCL actually picks:

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=GRAPH,INIT

OOM during weight loading

Cause: Another process is holding VRAM. Fix: Kill rogue processes and retry:

nvidia-smi
ps aux | grep vllm
pkill -f vllm

Node 2 connects but model output is incorrect

Cause: NCCL version mismatch between nodes. Fix: Confirm identical versions on both nodes:

dpkg -l | grep -E "nccl|cuda"

Conclusion

With the minimum 16×H100 configuration on GreenNode's rail-optimized InfiniBand fabric, serving a 1T parameter model is a solved problem. The hard parts – NCCL configuration, fp8 inference, multi-node parallelism strategy – are validated and documented above.

If you hit issues not covered here, reach out to the GreenNode team. We run this stack in production and are happy to help you get unblocked.

Deploy 1T params model with minimum hardware on GreenNode