Introduction

A 2.8 trillion parameter model shouldn't be something a startup can self-host. And yet, as of this week, that's exactly the pitch.

On July 27, 2026, Moonshot AI released the open weights for Kimi K3, a Mixture of Experts model that Moonshot describes as the first open-weight system to cross the 3-trillion-parameter class. Three days later, AWS published a deployment guide for running it — not one path, but two: Amazon SageMaker HyperPod and Amazon EKS. That's a fast turnaround for infrastructure documentation, and it says something about how AWS is positioning itself in the open-weight arms race: not as a gatekeeper, but as the plumbing.

The more interesting question isn't whether you can run Kimi K3 on AWS. It's what running it actually costs you in hardware, complexity, and operational commitment — and whether that changes how enterprises think about self-hosting frontier-scale models at all.

What Actually Changed With Kimi K3

Kimi K3 isn't just bigger than its predecessor, Kimi K2 — it's structured differently. The model distributes its parameters across 896 specialist experts, but only activates 16 per token.

AttributeValue
Total Parameters2.8 Trillion
Active Parameters per Token104 Billion
ArchitectureMixture of Experts (MoE)
Expert Count896 (16 activated per token)
Context Window1 Million Tokens
ModalityNative Multimodal (Text + Vision)
Release DateJuly 27, 2026

That sparse activation pattern is the point. Only about 104 billion of the 2.8 trillion parameters are doing work on any given forward pass, which is why Moonshot claims a 2.5x improvement in scaling efficiency over Kimi K2. The architecture — built on Kimi Delta Attention, Gated Multi-Head Latent Attention, and what Moonshot calls a Stable LatentMoE framework — is designed specifically for long-horizon coding and multi-step agentic workflows, the kind of tasks where a model needs to hold context and intent across dozens of tool calls, not just answer one prompt well.

"The weights are distributed in MXFP4 (Microscaling Floating Point 4-bit) format, which provides an effective balance between model quality and memory efficiency for large-scale inference deployments."

That quantization choice matters more than it sounds. It's the difference between a model that requires a research lab's GPU budget and one a well-resourced engineering team can plausibly run.

Why This Requires Purpose-Built Hardware

Here's the part that keeps this from being a weekend project: Kimi K3 requires a p6-b300 instance — ml.p6-b300.48xlarge — which packs 8 NVIDIA B300 Blackwell Ultra GPUs with high-bandwidth interconnects. That interconnect isn't optional. Tensor-parallel inference across 896 experts needs GPUs that can talk to each other fast, or the "16 experts per token" efficiency gain gets eaten by communication overhead.

Why This Matters: Unlike smaller open-weight models that run on a single consumer or prosumer GPU, Kimi K3 requires reserved, multi-GPU capacity from day one. That reframes the deployment decision from "which framework do I use" to "how do I procure and reserve enough Blackwell-class compute."

AWS offers two ways to secure that capacity:

  • Flexible Training Plans — committed capacity reservations allocated directly to a SageMaker HyperPod cluster.
  • Capacity Blocks — time-bound EC2 GPU reservations, aimed at EKS workloads that need guaranteed access without a long-term contract.

Which one an organization picks says a lot about how they plan to operate. This is where the two deployment paths diverge.

Path 1: SageMaker HyperPod — Managed Simplicity

For teams that want to minimize orchestration overhead, AWS points to SageMaker HyperPod with its Inference Operator, which installs automatically during cluster creation and handles container scheduling, model loading, and endpoint health checks.

The setup, at a high level:

  1. Create a HyperPod cluster orchestrated by Amazon EKS, using either Quick Setup or a custom VPC configuration.
  2. Add a worker instance group using ml.p6-b300.48xlarge.
  3. Procure capacity through a Flexible Training Plan and set the target Availability Zone to match.
  4. Apply an InferenceEndpointConfig manifest that points to the Hugging Face model ID moonshotai/Kimi-K3 and a day-0 vllm/vllm-openai:kimi-k3 serving container.

The deployment manifest itself is where the model-specific details live — tensor-parallel-size set to 8 to match the GPU count, the MXFP4-aware fastsafetensors load format, and a Kimi-specific tool-call and reasoning parser. Once the endpoint reaches a ready state, it exposes a standard OpenAI-compatible /v1/chat/completions route.

The trade-off is control for convenience — HyperPod abstracts away the Kubernetes plumbing entirely, which is exactly what teams without dedicated infrastructure engineers need.

Path 2: Amazon EKS — Full Control

Teams that already run their own Kubernetes infrastructure, or want more granular control over networking and scaling policy, can deploy on a standalone EKS cluster instead, reserving p6-b300.48xlarge capacity through EC2 Capacity Blocks.

Kimi K3 on EKS: HyperPod vs. Self-Managed — HyperPod trades flexibility for a managed operator; EKS trades a heavier setup lift for full ownership of the stack.

AWS's ai-on-eks project provides Terraform modules covering the full path: VPC and node group provisioning, NVIDIA device plugin installation, a Helm-deployed vLLM server with the same tensor-parallel and MXFP4 arguments used in the HyperPod config, and a LoadBalancer or Ingress-exposed endpoint. The serving arguments are functionally identical across both paths — the difference is entirely in who manages the orchestration layer underneath.

Invoking the Model

Regardless of which path a team chooses, the resulting endpoint speaks the same language: an OpenAI-compatible chat completions API.

from openai import OpenAI

client = OpenAI(
    base_url="http://<ENDPOINT_URL>:8000/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="Kimi-K3",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the benefits of mixture of experts architectures."}
    ],
    temperature=0.7,
    max_tokens=1024
)

That compatibility is deliberate. It means existing tooling built around OpenAI-vs-Claude comparison stacks doesn't need to be rewritten just to point at a self-hosted model — only the base URL changes.

Why Now?

This didn't happen in a vacuum. Open-weight models have been creeping toward frontier-model territory for two years, but the infrastructure to actually run them at that scale has lagged behind the weights themselves. A 2.8-trillion-parameter MoE model with a 1-million-token context window isn't something most companies could serve reliably even a year ago — the GPU generation didn't exist, and neither did the tooling to orchestrate hundreds of experts across nodes efficiently.

Bottom Line: AWS shipping a same-week deployment guide, with a day-0 vLLM container built specifically for Kimi K3's architecture, signals that cloud providers now treat frontier-scale open-weight releases as launches to support immediately — not months later.

What Changes for Enterprises

For organizations evaluating whether to self-host versus use a hosted API, Kimi K3's AWS support shifts the calculus in a specific way: the ceiling on "self-hostable" model size just moved. Compliance-sensitive industries — finance, healthcare, government contractors — that need models to stay inside their own VPC now have a genuinely frontier-capable option, not just a smaller open-weight fallback.

That doesn't make it cheap. Reserved p6-b300.48xlarge capacity, whether through a Flexible Training Plan or a Capacity Block, is a meaningful commitment before a single inference request is served. This is infrastructure for teams running sustained, high-volume agentic or coding workloads — not a side project.

What Happens Next

The vLLM support for Kimi K3 is still landing in a dedicated vllm/vllm-openai:kimi-k3 branch rather than the main container, with AWS noting that a merge into mainline vLLM is expected in upcoming releases. Once that happens, deployment friction should drop further, since teams won't need to track a model-specific container tag.

The bigger open question is capacity. Both deployment paths depend on reserved Blackwell Ultra GPU availability, and demand for p6-b300 instances is likely to climb as more teams attempt exactly this deployment. Whether AWS can keep that capacity accessible without long queues will shape how many organizations actually follow through on self-hosting versus falling back to a hosted API.

FAQ

What GPU instance does Kimi K3 require on AWS? Kimi K3 requires the ml.p6-b300.48xlarge instance, which provides 8 NVIDIA B300 Blackwell Ultra GPUs.

What's the difference between the SageMaker HyperPod and EKS deployment paths? HyperPod uses a managed Inference Operator that handles orchestration automatically, while EKS gives teams full control over their own Kubernetes stack, provisioned through Terraform and Helm.

Does Kimi K3 use an OpenAI-compatible API? Yes. Both deployment paths expose a standard /v1/chat/completions endpoint, so existing OpenAI SDK-based tooling works with only a base URL change.

What format are the Kimi K3 weights distributed in? MXFP4 (Microscaling Floating Point 4-bit), chosen to balance model quality against memory efficiency at this scale.

Key Takeaways

  • Kimi K3 is a 2.8 trillion parameter MoE model with 104 billion active parameters per token, released by Moonshot AI on July 27, 2026.
  • AWS supports two deployment paths: managed SageMaker HyperPod and self-managed Amazon EKS, both requiring ml.p6-b300.48xlarge GPU capacity.
  • Both paths use a day-0 vLLM serving container and expose an identical OpenAI-compatible API.
  • Reserved GPU capacity — via Flexible Training Plans or Capacity Blocks — is a prerequisite, not an afterthought, for either path.
  • The deployment marks a shift in what "self-hostable" means for frontier-scale open-weight models.

Suggested External Sources