Accelerated Infrastructure: Building a Backbone for AI — The Futurum Group at Marvell’s AI Day

Disclaimer: The Futurum Tech Webcast is for information and entertainment purposes only. Over the course of this webcast, we may talk about companies that are publicly traded, and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors, and we ask that you do not treat us as such.

Transcript:

Daniel Newman: Hey everybody. Welcome to the Futurum Tech Podcast. I’m Daniel Newman, CEO of The Futurum Group here in beautiful Santa Clara by the bay at Marvell’s California main office. They don’t call them headquarters anymore, but you know what? That’s because the company’s expanded so much. It has presence all around the world. I’m going to be joined today… We’re going to be talking about generative AI. We’re going to talk about a lot more, but I’m talking to Raghib. Raghib, welcome to the show. How are you?

Raghib Hussain: Thank you. I’m doing pretty good.

Daniel Newman: Hey, you did a great keynote this morning. Appreciated you getting up there and talking to us. It’s been a year. I did really love Chris’s slide that showed one year ago and six days when Sam Altman tweeted about basically, “Check out our new thing, ChatGPT.” Think about a year. Now, if you look at the curve, the law of diffusion of innovation, you look at transformation from the printing press to the telephone and the generation to generation evolutions are huge. Now in a year, we’ve seen 7,000 iterations, 500 new competitors. I’d love to get your just take on the overall what’s happened in the last year and what’s going on with generative AI.

Raghib Hussain: Yeah, as you correctly said, this whole world has pretty much changed in the last one year. What has happened in one year is generally like a decade or maybe more because the technology transformation takes time. But that is a main point about the generative AI. It is causing so much disruption in the overall infrastructure, in the overall world, I would say the way things will be done. And the need, the hunger for performance is driven by this, so it is going to disrupt pretty much every aspect of the way infrastructure are built because the focus on the performance and efficiency is there.

Daniel Newman: Are you using it at all? Any tools?

Raghib Hussain: Oh, in my personal life, I actually not only I use myself and I like the voice interface in ChatGPT, so I use it a lot, but I also encourage my kids to use it. Before they were using more like a search engine, I said, “No, no, no, no. Establish a relationship with them.” They say, “What do you mean by establish a relationship?” I say, “You know what? Just talk. Anything you want to talk, just talk.” And now I think they’re getting it where I made it mandate, any question you want to ask anybody, first ask ChatGPT, then ask anybody else, and it’s kind of an eyeopener for them.

Daniel Newman: Well, it’s really interesting to see, I mean obviously things like empathy and the deep human condition is going to be a little harder to emulate. Although AGI and stuff, I don’t know, it could get pretty interesting. But what I will say is in terms of information flow, the rapid improvement of its ability to sort of multi turn and create these continuous interactions of high quality feedback. It’s been really impressive. And by the way, not just ChatGPT, I mean there are so many options. You look at all the models, all the tools. You want it written, you want images.

Raghib Hussain: That is true.

Daniel Newman: It’s been incredible, but it’s been a big driver of business for Marvell. You were so well positioned in this particular space. And to some extent, I mean you knew it, but you almost didn’t realize just how well positioned you were.

Raghib Hussain: Yeah. This is one of those things which we always believe that data is going to drive the needs of the infrastructure. And we always believe, and we could see data is growing, everybody’s generating so much data, and that’s what we talked last year and that is why we believe to achieve the full value out of this data, you’ll need much more accelerated infrastructure. Now, of course, we have been building these capabilities for a long time, but now we are in the middle and center of this thing where AI actually driving the need of infrastructure so much so that everything has to accelerate at a much faster pace.

Daniel Newman: Yeah. So talk a little more about that, Raghib. Talk about accelerated infrastructure. That’s been the big highlight of this event. Of course, you’re moving towards cloud optimized silicon for AI. There’s going to be that. That’s coming down the pipe for Marvell, but the accelerated infrastructure is here today and the data center, it’s been like a double every quarter in terms of revenue for the past several quarters. Everyone’s asking, “How fast can it go?” You have to build, and you’re kind of answering that, if not directly, indirectly by continuing to optimize. So what is this accelerated infrastructure?

Raghib Hussain: Yeah. If you look at all these models needed for AI, these are really complex models, but the intelligence, the value of this model, depending on how many parameter it has and how complex of a good relationship between those parameters. To achieve that, you really need a different scale of compute capability. Not only a single engine compute, single server compute capability, but also these models are very large in size, do not fit in a single AI processor or GPU memory. It is spread over multiple AI server. In fact, it is spread over multiple nodes in the network. Then you need to process it in parallel. As a result of that, you need to move a lot of data back and forth. That is why the performance of how quickly can you train these model or how quickly can you extract inference out of or value out of this model depends on the performance of the entire infrastructure. That is why, as we explained earlier today, the infrastructure which is being built to serve the need of these AI processing are completely different than a traditional data center. That’s why these are called the accelerated infrastructures. This is where we are actually in the front and center of this. There are two component of this accelerated infrastructure. One is compute. You need a specialized compute, either GPU or a specialized compute for AI processing, which is custom silicon where we work with the hyperscalers. The other very critical part, and I should say that the value in these two pillars of this accelerated infrastructure is the interconnectivity. So the connectivity like a high bandwidth, low latency switches, as well as optical and copper interconnect. And Marvell actually has been investing in developing those capabilities for years. In fact, we have 20 plus years experience in data centric compute as well as connectivity.

Daniel Newman: Yeah. We’ve been talking a lot about, does this mean the whole data center game changes? And I think this year we’ve seen the gold rush has been all about data center hardware and infrastructure. There’s kind of like this AI boom, and Marvell has been a beneficiary because you have certain infrastructure that’s required. There’s been a very small number of other companies, and most people know who I’m talking about that have been the outsized beneficiary of this. And then over time, I think you’ll see it’ll change a bit as we see more inference and training, maybe the exponential inference and the more linear training growth. But there’s an argument that smaller models will create an exponential growth of training as well. But you seem to be in the right place. It seems that there’s a requirement for new data center architectures almost from the ground up. Talk a little bit about how Marvell is enabling this new architecture. What are you offering to the market to make this happen?

Raghib Hussain: Yeah, definitely. So as I said, you need this specialized infrastructure, what we call the accelerated infrastructure. And of course there are two critical pillar of these infrastructure, compute and connectivity. When you look at the various architecture, there are two types of network architecture out there. One are those data center based on the GPU based compute. The other one is the custom AI silicon based compute. But if you look at the overall architecture, whether it’s a GP compute or the custom AI compute, they needs to have a very high bandwidth low latency network. So Marvell actually is a leader in providing those optical interconnect, which they’re called the PAM-4 interconnect is like a right now, 800 gig type of connectivity they provide and going to one 1.60 to 3.2 T for the connectivity.

Daniel Newman: Which is fast.

Raghib Hussain: That is really fast and it is doubling very fast also. The cadence of improvement is also very fast. And then there are also active ethernet cables, ACs, which is like a copper connectivity, but for the shorter range distance, you can use those. So Marvell has those products too. In addition, there are lots of things around it. So for example, when you have a GPU or AI computer, there are the pre-processing and post-processing needed, and there are value of how you connect the various memories or storage with the system because at the end of the day, only compute and connectivity is not going to solve your problem. You need to connect memory, you need to do pre-process.

And Marvell has products in all those areas as well, especially we have a very huge investment and initiative going on in CXL, which allows near-memory compute, which is more of a pre-processing for these type of solution. So if you look at it, we have a very comprehensive solution for all kind of connectivity, including optical interconnect, including switches, because these switches are also, you need high bandwidth switch with a low latency to connect these optical interconnect. So we have that. We have every aspect of the connectivity as well as we play in the compute side by partnering with our hyperscale customers who are building their own custom compute. This is where our partnership actually plays a critical role.

Daniel Newman: Yeah. As we sort of wrap up here, I really think it’s important, and I sometimes think the market doesn’t fully appreciate this, is you’re building a lot of capabilities that you’re telling and then you’re also enabling a lot of companies. We talk to your COO, Chris Koopmans and talked about ecosystem and the number of companies that turn to Marvell as their partner to develop these custom silicon off the shelf capabilities so that they can participate in this AI data center growth is significant. We’re talking about big important companies. Some of these companies that are the ones that are announcing these next generation, it’s with Marvell at their side. I think that’s a really important footnote, Raghib, because you guys, you’re doing a lot. And by the way, you can have all the GPU power in the world, but if you can’t move the data, if you can’t make it move fast enough, these cool apps and these workloads and these next generation enterprise tools, they don’t come to fruition.

Raghib Hussain: Yeah, definitely. I mean, think of it as if GPU or AI computed like a brain, the whole network is a body, like a whole connectivity. And you all know, especially without the body, brain is not going to do much, right? So yeah, while brain is very important, but it’s very critical to have the right size of the rest of the network as well. This is where we are enabling all kind of architectures out there to make it possible. Also, as I said, the speed of improvement, the cadence has increased so fast that no single company can do it all themselves. That is where the value of partnership comes. We are partnered with the GPU companies. We are also partnered with the hyperscalers to build their own custom compute, and we are also partnering with other ecosystem to bring the improvements, let’s say, in packaging, which is a very complex technology now to make these complex chip possible or module vendors and so on and so forth.

Daniel Newman: Absolutely. At times, you are a well kept secret, but in many cases when you’re seeing these next generation architectures, Marvell had a part to play.

Raghib Hussain: We are enabling. As I said, we are in front and center of this whole revolution, which is going on.

Daniel Newman: Raghib, thanks so much for taking a little time to talk about this, and congratulations on all the progress.

Raghib Hussain: Thank you.

Daniel Newman: All right. Thanks everyone for tuning into the Futurum Tech Podcast. We’re talking about the future of data center architecture and accelerated infrastructure and computing here on the show. Hit that subscribe button if you like what you heard. Hit that subscribe button anyway. We hope you’ll stay with us for all our episodes, but for now, I got to say goodbye. We’ll see you all later.