OpenAI's Custom Chip: A Game-Changer for AI Inference

OpenAI has unveiled its first custom-built inference processor, designed and manufactured in collaboration with Broadcom. This new chip, named Jalapeño, is specifically designed for the unique needs of OpenAI's inference systems and has shown significantly better performance-per-watt than current state-of-the-art alternatives.

A Break from Nvidia GPUs

The partnership between OpenAI and Broadcom was officially announced in October, but the company's plans to build custom chips have been long-rumored as a way to reduce its dependence on Nvidia's GPUs. Google and Amazon have also built custom chips to serve a similar purpose, often called 'AI accelerators' - silicon designed specifically to speed up machine learning workloads.

OpenAI's Approach to Chip Development

OpenAI president Greg Brockman explained the company's approach to chip development on its in-house podcast, shortly after the Broadcom partnership was announced. Brockman said, "We have a deep understanding of the workload. We've really been looking for specific workloads that are underserved, [and asking] how can we build something that will be able to accelerate what's possible?"

Jalapeño's Design and Benefits

Jalapeño is specifically designed for inference, the process of running pre-built AI models in response to user commands. In the announcement, OpenAI emphasized the chip's low operating cost when running real-time coding models. It's likely that more performance-intensive tasks like pre-training will still rely on Nvidia hardware, but even small reductions in inference costs could do a lot to improve the company's bottom line.

• Optimizing inference systems may prove crucial in the economics of AI going forward.
• OpenAI is already building agentic products like Codex and the models that power them, as well as data centers to run those models.
• Moving into purpose-built chips lets the company go even further in that process.

OpenAI's Stack Optimization

"OpenAI is not only developing frontier models or building products on top of them; it is designing the infrastructure underneath them: chip architecture, kernels, memory systems, networking, scheduling, deployment systems, and product experience," the company wrote. "Because OpenAI operates across the stack, each layer can be optimized around the same goal: making its models faster, more reliable, and more affordable for users."

When you read through this article, you'll notice that OpenAI is taking a comprehensive approach to AI development, from building custom chips to designing infrastructure and products. This level of optimization is expected to benefit the company's bottom line and improve the overall performance of its AI models.