Analyst(s): Ron Westfall
Publication Date: December 16, 2024
What is Covered in this Article:
- New Marvell AI accelerator (XPU) architecture enables up to 25% more compute, and 33% greater memory while improving power efficiency.
- Marvell collaborated with Micron, Samsung, and SK hynix on custom high-bandwidth memory (HBM) solutions to deliver custom XPUs.
- Architecture comprises advanced die-to-die interfaces, HBM base dies, controller logic, and advanced packaging for new XPU designs.
The News: Marvell Technology Inc., a supplier of data infrastructure semiconductor solutions, announced that it has developed a new custom HBM compute architecture that can enable XPUs to achieve greater compute and memory density.
Marvell HBM Compute Architecture Ready to Transform Cloud AI Acceleration
Analyst Take: Marvell deftly leveraged its annual Marvell Industry Analyst Day to debut its custom high bandwidth memory (HBM) compute architecture proposition. Marvell’s new HBM compute architecture technology is available to all of its custom silicon customers to improve the performance, efficiency, and total cost of ownership (TCO) of their custom XPUs. Marvell is collaborating with its cloud customers and HBM manufacturers, Micron, Samsung Electronics, and SK hynix to define and develop custom HBM solutions for next-generation XPUs.
The Marvell custom HBM compute architecture introduces tailored interfaces for XPU designs, optimizing performance, power, die size, and cost. This approach considers the compute silicon, HBM stacks, and packaging, moving beyond the limitations of current standard interface-based architectures.
Marvell’s Custom HBM Portfolio Advantages
By customizing the HBM memory subsystem, including the stack itself, Marvell is advancing customization in cloud data center infrastructure. The company’s collaboration with major HBM manufacturers promises to accelerate the implementation of this new architecture and fulfill the most pressing needs of the hyperscalers.
At the Marvell IAD event, Marvell Technology Senior Vice President and General Manager, Custom Compute and Storage Will Chu spotlighted why HBM innovation is integral to advancing XPU evolution and why Marvell delivers competitive advantages. Custom HBM (cHBM) designs allow for optimizing memory and memory controllers to closely match the requirements of processors or AI accelerators, enabling tweaks in bandwidth, latency, and memory configuration. The outcomes are compelling in relation to standard JEDEC HBM implementations and are ready to move the needle in XPU optimization:
- Higher performance die-to-die (D2D) I/O
- 70% lower interface power
- 25% more XPU silicon area
- 33% more HBM stacks
- Improved TCO
I find that Marvell’s HBM customization portfolio initiative is key to driving XPU innovation due to the ability to improve design alignment with the specific needs of XPU makers. The JEDEC standards have played a crucial role in ensuring the interoperability, reliability, and performance of electronic components including HBM memory chips, microprocessors, and other integrated circuits.
However, Marvell is showing why cHBM is essential for the next generation of XPU design and development, especially across hyperscaler environments as hyperscalers have significantly increased their capital expenditure by approximately $100 billion in 2024.
The upcoming generation of hyperscale AI clusters is projected to be at least 10 times larger than the 100,000 GPU xAI Colossus Cluster, potentially consuming several gigawatts of power. As these clusters expand, even minor power savings can produce megawatt-scale reductions.
The Move from JEDEC Standard HBM to Custom HBM
The shift toward customized memory solutions for hyperscalers, moving away from JEDEC standards, is a game-changing development. This transition indicates that Marvell offers the portfolio capabilities key to driving hyperscale XPU innovation and beyond.
Marvell’s cHBM initiative addresses the memory bottlenecks that impede standard common HBM technology. Custom HBM, when combined with D2D interfaces, helps tackle the memory bottleneck problem faced by AI chips, crucial for handling massive data requirements of modern AI workloads.
The latest iteration of High Bandwidth Memory, HBM4, traditionally requires more than 2,000 pins, doubling the pin count of its predecessor, HBM3. However, cHBM eliminates the necessity for such a vast number of pins, resulting in a more efficient design. Moreover, this reduction in pin requirements liberates valuable die area on the chip. The newly available space can be utilized to integrate additional custom logic, such as enhanced compression algorithms and advanced security features, further improving the overall functionality and performance of the memory system.
Through its cHBM architecture, Marvell is now deftly positioned to meet the rapidly growing demand for more memory as this trend puts pressure on designers to economize on space, power, and cost. HBM currently accounts for 25% of the available real estate inside an XPU and 40% of the total cost. With cHBM, XPU makers can attain total cost of ownership (TCO) breakthroughs that are not possible with standard JEDEC capabilities.
The competitive advantages of the new Marvell XPU architecture demonstrate that cHBM implementations can lead to better power efficiency and smaller form factors, which are critical for AI systems deployed at scale. Customization allows fine-tuning the memory system to meet the demands of specific AI and high-performance computing applications, such as deep learning training and inferencing.
Looking Ahead
The advent of cHBM marks a significant milestone in memory technology. However, this innovation is just one facet of a broader, industry-wide shift. As Moore’s Law reaches its limits, the tech sector is increasingly turning to custom compute design as the new frontier for enhancing performance and capabilities.
I anticipate that Marvell will also enhance integration flexibility throughout custom HBM implementations by also supporting UCIe D2D interfaces. Such support can enable tighter integration between memory dies and compute dies, which can achieve advances in higher bandwidth and low latency outcomes. Moreover, customization is also extending beyond XPUs and CPUs to CXL controllers, NICs, and other devices.
By enabling these advantages, Marvell HBM customization solutions can play a crucial role in driving XPU innovation, particularly for AI and high-performance computing applications that require extreme memory bandwidth and efficiency. Marvell’s adoption of a comprehensive strategy for tailored chip design, which scrutinizes each component for possible enhancements, will be crucial in making AI both environmentally sustainable and cost-effective in the long run.
See the complete Marvell press release on the Marvell site.
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Other Insights from The Futurum Group:
Marvell Right Sizes AEC Connections to Meet New AI Acceleration Demands
Marvell Unveils Structera CXL Solutions to Meet Hyperscaler Memory Needs
Marvell Q3 FY2025: AI & Cloud Growth Propel Revenue Amid Restructuring
Author Information
Ron is an experienced, customer-focused research expert and analyst, with over 20 years of experience in the digital and IT transformation markets, working with businesses to drive consistent revenue and sales growth.
Ron holds a Master of Arts in Public Policy from University of Nevada — Las Vegas and a Bachelor of Arts in political science/government from William and Mary.
