700 Tokens Per Second: How Google TPUs Just Entered Meta's Production Stack

Google’s custom silicon is now running critical AI workloads in production at Meta, a development that marks the first time the company’s Tensor Processing Units have been sold to an external hyperscale customer. This milestone was confirmed through the release of a new, PyTorch-native version of the SGLang inference engine. For years, Meta has been heavily associated with the PyTorch framework, while Google’s hardware was tightly coupled to its own JAX framework. By removing that barrier, Google has made its TPUs a native option for the industry's standard deep learning library, largely eliminating the need for code rewrites.

The strategic significance of this move lies in its potential to disrupt Nvidia's market position. Nvidia has long maintained a dominant share of the AI hardware market, partly due to a deep software moat that makes its ecosystem difficult to leave. By providing a seamless bridge for PyTorch, Google and Meta are signaling that companies no longer need to commit to a single vendor's software stack. This flexibility allows Meta to deploy its models on Google hardware without abandoning its existing infrastructure, directly challenging the notion that Nvidia is the only viable option for large-scale AI training and inference.

Zongwei Zhou, Meta’s lead for PyTorch on TPUs, stated that the new SGLang-TorchTPU is already live in Meta’s production environments. He credited the success to a close collaboration between Meta, Google, and the SGLang team. Zhou noted that the engine provides Meta with a remarkably mature serving stack that is fully integrated with PyTorch. This team operates out of RadixArk, a startup founded by the creators of SGLang, which developed the engine in partnership with both tech giants.

Jana van Greunen, director of PyTorch engineering at Meta, emphasized the company's need for hardware independence. She stated that Meta cannot be constrained by a single hardware architecture. According to Greunen, the new TorchTPU backend enables the company to deploy its PyTorch workloads directly onto Google TPUs. Google confirmed that it co-developed this integration with Meta’s engineering team, highlighting a shift from a purely commercial relationship to a deeper technical partnership.

This is not the first time Meta has experimented with Google hardware. The company previously used TPUs through an older bridge called PyTorch/XLA. Reports indicated that Meta research teams utilized this setup from 2020 to 2023 before canceling those contracts. The current push appears more permanent, with recent talks involving Google Cloud TPU rentals in 2026 and potential hardware purchases in 2027. Meta is also reported to have signed a multi-year rental agreement, suggesting a long-term commitment to this infrastructure.

The timing of this announcement is notable, as it came one day after Amazon open-sourced TorchNeuron, a similar backend for its own Trainium chips. Both moves appear designed to loosen Nvidia's software lock-in. By providing native PyTorch support for their respective hardware, Google and Amazon are giving large AI labs the option to diversify their compute resources. This trend suggests a future where major AI companies can mix and match hardware vendors without significant software overhead.

Performance metrics support the viability of this new setup. A tuned vLLM configuration serving the Kimi K3 model, which has 2.8 trillion parameters, reportedly achieves over 700 tokens per second for a single user on Google's seventh-generation Ironwood TPU. This figure comes from Inferact, a startup involved in building the vLLM-TorchTPU integration. In comparison, the same model running on 16 Nvidia GB200 chips reached 452 tokens per second. The new kernels for Kimi K3 and Qwen 3.8 27B deliver roughly 1.4 to 2 times the throughput of the Nvidia baseline at small batch sizes, even without speculative decoding.

The ecosystem is expected to expand rapidly. SGLang-TorchTPU will continue adding model support, starting with DeepSeek V4.1 Flash and Reflection Beam. The teams are aiming for Day 0 TPU support for new open models by the first quarter of 2027. Google, Meta, and RadixArk plan to share further details at the SGLang Summit 2026, scheduled for November 12 and 13 in San Francisco. This continued collaboration suggests that the integration of Google silicon into non-Google AI stacks is just beginning.

For California-based tech readers, this shift signals a more competitive landscape for AI infrastructure. The ability to run high-performance inference on alternative hardware could lead to lower costs and greater flexibility for local startups and enterprises. As the software barriers between major cloud providers erode, the focus will increasingly shift to raw performance and price, benefiting the broader AI ecosystem.