The Six Five team discusses Marvell Structera to solve memory challenges for cloud providers.
If you are interested in watching the full episode you can check it out here.
Disclaimer: The Six Five Webcast is for information and entertainment purposes only. Over the course of this webcast, we may talk about companies that are publicly traded and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors and we ask that you do not treat us as such.
Transcript:
Daniel Newman: Let’s talk about Marvell and their new Structera product, which is all about solving memory challenges for cloud using CXL?
Patrick Moorhead: Yeah, let’s jump in baby. Yeah, I think sometimes you use long post warning. This is going to be a deep dive warning here. First thing I need to do is give kind of a backdrop of some things that are going on in the infrastructure world. And we know infrastructure is cool. So, CPU cores are going up, but memory bandwidth per core and per socket is going down. And what happens is you get idling, right? You’re not getting the full use of it. And it’s hard to have giant memory planes to be able to put enough memory to be able to serve those cores. And this is important in almost every application. But stuff like AI inference and in-memory databases is becoming true. And so just some examples like AMD EPYC has 256 cores, Genoa, I think the code name is Intel Sierra Forest 288 cores.
AmpereOne, they just announced, well, they announced 256 cores. So pretty crazy. The first server … I hired the first AMD product manager at AMD, one core. And that might give you an idea of how crazy that was. That was 20 years ago. So, enter a new technology that has been in the works for, gosh, five, six, seven more years called CXL, and CXL stands for Compute Express Link. And there’s different versions of it, huge consortiums that have gone in or do that. And CXL 2.0 allows memory pooling and network switching. Think of one giant planar memory surface. Think of using memory like we do storage where you can just add a ton of it in a rack. And think of there’s this memory and storage pool. Think of CXL bridging the gap between memory and flash. So you can think of memory per socket over three terabytes, memory bandwidth a hundred gigabytes per second.
Okay, I’m getting here, okay, that was the groundwork. Marvell brought out three chips across two families under the brand of Structera. Hopefully I’m saying that right. Structera A is all about scaling, compute and memory bandwidth. Some real interesting case study they did in their presentation in the press release about recommendation engines. They didn’t call out Meta or Amazon. But when you think about serving ads and serving up products for people to buy or friends to follow, that is all about a recommendation engine. You want very tight coupling between the CPU and even the inference GPU. I think of Meta and I think of Amazon here and more.
The wild part, this isn’t just an accelerator chip, it has 16 Neoverse V2 cores. That’s big cores for all those out there, 100 watts, five nanometer. Second product line is Structera X, this is all about memory capacity. There’s two versions of this, one for DDR4, one for DDR5, 30 watts. The cool part is like, “Well, why DDR4?” You can take all of your investment that you had in DDR4 in DIMMs, plop this into a Structera X-capable server, and boom, you can pool and reuse all that memory. And of course, it supports DDR5 as well. So, other players in this CXL storm, Samsung, Semi, Micron, Astera Labs, Broadcom with Retimers. It’s cooking. By the way, this is just one of these things that we sometimes take for granted, this is CXL has been in play for so many years, but it’s getting more real, folks.
Daniel Newman: Wow, that was super nerdy. But this is the area where a company like Marvell that’s really been focused on accelerated infrastructure. So, what does that mean? Well, yes, they’re one of the two XPU companies. When you’re hearing about these various companies building AI accelerators or specialty AI chips, it’s going to be the Broadcom or it’s going to be Marvell in the background. And accelerated infrastructure is going to be critical, because it’s really dealing with everything from like you just talked about, CXL memory pools and being able to get more out of your … The limited capacity and availability of high-bandwidth memory to being able to create network efficiency, power efficiency. When they’re talking about accelerated infrastructure, they’re not just talking about switching, but they’re really talking about how to maximize and create great levels of efficiencies and get the most performant infrastructure as AI is going to continue to have really challenging economics and challenging power envelopes on the entire grid, on the entire world.
So, I think you hit the technical stuff pretty well. I kind of think about it. I used a really simple analogy, fact from my earlier days in my career when I was in video compression. But the whole idea of, I don’t if you remember H.264, but the ability to take twice the quality, same amount of bandwidth, or same quality half the bandwidth. The idea though of trying to figure out how to get more out of the same amount of memory is going to be a problem that companies are going to try to solve. It sounds like that’s what CXL, that’s what they’ve been trying to do. I would say that’s interesting. This has been a bit of a stop start trend line. It’s like, we’ll hear about it a bunch and then I don’t hear about it.
I remember about two years ago, I feel like I did five or six different press interviews all about what CXL and what’s it going to do and how’s it going to change. And then I don’t feel like I’ve heard anything and then all of a sudden it’s kind of popping back up. But it makes a lot of sense right now with HBM. And of course, we know right now that this is the, we talk about the limiting factor being … CoWoS and the limiting factor being amount of process and capacity with TSMC and others. But HBM is another limiting factor of AI. It just is. You can only make so much if you don’t have enough. So, dealing with this, figuring out technologies that enable us to get more out of the bandwidth that we have, the memory that we have is going to be important. So, I’m not going to nerd out any more.
Patrick Moorhead: Oh, and by the way, it wouldn’t be Pat Moorhead without talking about the infrastructure quadrangle, or as you say, the infrastructure square. And that is, as technology moves forward we need to make sure compute, storage, memory and networking are all aligned, otherwise it’s hard to get any innovation without doing some weird things like HBM.
Author Information
Daniel is the CEO of The Futurum Group. Living his life at the intersection of people and technology, Daniel works with the world’s largest technology brands exploring Digital Transformation and how it is influencing the enterprise.
From the leading edge of AI to global technology policy, Daniel makes the connections between business, people and tech that are required for companies to benefit most from their technology investments. Daniel is a top 5 globally ranked industry analyst and his ideas are regularly cited or shared in television appearances by CNBC, Bloomberg, Wall Street Journal and hundreds of other sites around the world.
A 7x Best-Selling Author including his most recent book “Human/Machine.” Daniel is also a Forbes and MarketWatch (Dow Jones) contributor.
An MBA and Former Graduate Adjunct Faculty, Daniel is an Austin Texas transplant after 40 years in Chicago. His speaking takes him around the world each year as he shares his vision of the role technology will play in our future.
