OpenAI published the first measured benchmark results for Jalapeño, its custom AI inference chip developed with Broadcom, showing the 700-watt processor outperforming Nvidia’s 1,400-watt flagship on efficiency metrics. The results, presented at Hot Chips 2026 this week, mark the first public evidence that a purpose-built inference ASIC can deliver meaningful gains over general-purpose GPU hardware at data center scale.
Across three open-weight models tested using OpenAI’s InferenceX benchmark framework, Jalapeño delivered 1.5 to 1.9 times more AI throughput per watt at peak performance compared to Nvidia’s GB200 and GB300 systems. End-to-end latency was cut to between 28 and 59 percent of Nvidia’s comparison systems, with single-token decode performance holding a consistent lead even against Nvidia’s multi-token processing mode. On a 1-trillion-parameter Kimi K2.5 workload, Jalapeño achieved roughly 1.9x the throughput per kilowatt against Nvidia’s 1,400-watt GB300.
Architecture Built Around the Data Bottleneck
Jalapeño was designed from the ground up for one specific workload: serving large language models at inference time. The chip pairs a single reticle-sized compute chiplet with 216 GiB of HBM4 memory delivering 15.4 TB/s of bandwidth, all within a 700-watt power envelope. At peak, the silicon delivers 13.4 PFLOPs of MXFP4 matrix compute. A 2,048-chip system scales to 27 EFLOP/s and 432 TiB of aggregate memory, with network bandwidth split at 600 GB/s for the local 128-chip domain and 200 GB/s globally.
The architecture targets what OpenAI identified as the core efficiency problem in LLM serving: at decode time, generating each token is memory-bandwidth-bound, meaning the chip spends most of its time moving model weights from memory rather than performing arithmetic. By co-designing the compute and memory layout around this bottleneck, Jalapeño achieves higher realized utilization than general-purpose GPUs that carry silicon and power overhead for workloads they never run.
Nine Months From Concept to Tape-Out
The development timeline is as striking as the benchmark numbers. OpenAI and Broadcom went from initial architecture concept in late 2024 to RTL freeze in 2025, a late-2025 tape-out, and working silicon running Codex in early 2026 with ChatGPT following shortly after. The entire concept-to-tape-out cycle took roughly nine months, a sprint that typically takes three to five years for high-performance custom silicon.
OpenAI attributed the speed to deep software-hardware co-development with Broadcom and Celestica, which handled rack integration, and the use of its own AI models to accelerate parts of the chip design and kernel optimization process. Optimized attention and mixture-of-experts kernels produced by the internal AI system ran 1.5x to 1.8x faster than expert-written implementations, validated end-to-end on-chip.
Deployment and Roadmap
Engineering samples are already running production workloads in the lab, including OpenAI’s GPT-5.3-Codex-Spark model at production target frequency and power. Initial small prototype deployments are planned for late 2026, with full-scale production ramping in 2027 and 2028 alongside Microsoft, which is reportedly set to absorb approximately 40 percent of the initial production run for Azure. The companies have committed to deploying 10 gigawatts of Jalapeño accelerators between late 2026 and 2029.
OpenAI framed Jalapeño as the first generation of a multi-chip roadmap rather than a one-off project. Generation 2 will target further improvements in performance per watt, while generation 3 aims for economical, low-latency serving by unlocking aggregate HBM bandwidth across multiple chiplets. Broadcom CEO Hock Tan described it as “just the beginning of a multi-generation roadmap.”
Critically, Jalapeño is inference-only. Training runs, which remain the most computationally intensive part of AI development, will continue to depend on Nvidia’s H100 and GB200 GPU clusters. The chip does not replace Nvidia for frontier model training; it captures efficiency gains on the inference side, which represents the larger volume of ongoing compute demand as AI models serve billions of daily requests across ChatGPT, Codex, and the API.
What It Means for Nvidia
The benchmark results put competitive pressure on Nvidia’s dominance in AI inference, though the threat is narrower than headlines suggest. Google already runs TPUs for its own inference workloads, Amazon deploys Trainium and Inferentia, and Microsoft has its Maia chip. OpenAI’s Jalapeño adds another major player to the custom silicon trend that is gradually reshaping the AI hardware market.
For the broader industry, the development signals that the era of GPU-only AI infrastructure is ending. As inference volumes scale and cost per token becomes the defining metric, companies with enough workload volume to justify custom silicon will increasingly build their own. The winners in this new landscape will be those who can solve the full stack from chip design to deployment at data center scale.
discussion