MC-2: Australian Based Inference · Matilda

Most teams do not really choose where inference runs. They choose an API endpoint, send requests to someone else's infrastructure, and inherit that provider's latency, pricing, availability, privacy, and data residency trade-offs.

MC-2 tests a different assumption: that Australian organisations can run modern open-weight models locally, at production-serving throughput, on dedicated bare-metal infrastructure in Australia.

We evaluated MC-2, Maincode's bare-metal AMD MI355X inference cluster in Melbourne, across long-context streamed workloads, model families, concurrency levels, and serving shapes. The goal was not to find one isolated tokens-per-second number. The goal was to validate whether local infrastructure can deliver serious inference capacity under the kind of load production systems actually create.

Benchmark setup

Serving stack: vLLM
Endpoint: private OpenAI-compatible streaming API
Input shape: 10,000 input tokens
Output target: approximately 1,500 output tokens
Metric: aggregate output tokens/sec at stated concurrency to replicate production inference workloads.

Models were served using their published checkpoint formats through vLLM, without speculative decoding, pruning, or provider-side black-box optimisations.

Qwen3.6

Qwen3.6 (35B A3B FP8) fits well into a replicated serving pattern. A single vLLM server gives a useful baseline, but the stronger production shape is horizontal replication: one vLLM replica per GPU, routed behind the same OpenAI-compatible endpoint.

c32single vLLM

741.45tok/s
c648 replicas

2,589tok/s
c1288 replicas

5,117.75tok/s
c2568 replicas

9,526tok/s

peak 10,000 tok/s

The important result is the shape of the curve. MC-2 scaled from a single-GPU baseline to nearly 10,000 aggregate output tokens/sec on one node under 10k-context streamed load.

Higher concurrency increases total serving capacity while reducing per-stream speed only moderately across the replicated runs. That is the tradeoff production workloads often need: many simultaneous streams, each still moving at usable speed.

Kimi K2.6

Kimi K2.6 is a much heavier serving target than Qwen3.6. It is a sparse MoE model with 1T total parameters and 32B active parameters per token, so the absolute throughput numbers are lower. The goal was to test whether MC-2 could run Kimi at noteworthy performance under the same long-context streamed workload and scale throughput as concurrency increased.

c32vLLM

485tok/s
c64vLLM

716tok/s
c128vLLM

973tok/s
c256vLLM

1,309.88tok/s

peak 1,400 tok/s

MC-2 scaled Kimi K2.6 from 485 output tokens/sec at c32 to 1,309.88 output tokens/sec at c256. Throughput increased at every concurrency step. The per-stream tradeoff is clearer on Kimi. As concurrency increased from c64 to c256, median per-stream speed moved from 11.46 tok/s to 5.26 tok/s while aggregate throughput nearly doubled. That result matters as Kimi is not a small model being run in a friendly benchmark shape. It is a large MoE model served through a private endpoint under long-context concurrent load, served at highly performant speeds on our own cluster here in Australia.

What this shows

The results from MC-2 shows that serious open-weight inference can run locally on dedicated Australian infrastructure, and that the right serving shape can deliver strong aggregate throughput under production-style load. We are not claiming MC-2 beats every API provider on per-request speed. We are showing that serious, high-throughput open-weight inference can run on dedicated Australian infrastructure, under long-context concurrent load, with local control over where the workload executes.

Australian organisations should not assume that high-performance inference must always run through offshore APIs. With dedicated bare-metal infrastructure, local serving stacks, and open-weight models, inference can be fast, local, and production-oriented while keeping control over deployment, data residency, and the serving path.

MC-2 is the foundation for that work at Maincode alongside Matilda: bare-metal AI infrastructure for serious inference workloads, hosted here in Australia.