On this episode of the Six Five, Daniel Newman is joined by Groq’s Mark Heaps, Chief Tech Evangelist, for a conversation on Groq’s significant progress in the realm of AI acceleration. This discussion delves into Groq’s successful Series D fundraising, strategic shifts, technological advances, and its mission to make AI more accessible.
Their discussion covers:
- The pivotal role of AI accelerators and GPUs in the tech industry and Daniel’s insights as an advisor and investor in Groq.
- Groq’s announcement of their $640 million Series D fundraising success and its impact on the company’s valuation.
- Groq’s remarkable technological strides, including the development of a new chip in partnership with Samsung, transitioning from 14-nanometer to 4-nanometer technology.
- The significant strategic pivots Groq has undertaken, such as major acquisitions and a shift towards offering cloud solutions, aimed at enhancing their market position.
- Groq’s effort to democratize AI by making their technology accessible and affordable, supporting open-source models, and providing free access to developers.
Learn more at Groq’s website.
Watch the video below, and be sure to subscribe to our YouTube channel, so you never miss an episode.
Or listen to the audio here:
Disclaimer: The Six Five Webcast is for information and entertainment purposes only. Over the course of this webcast, we may talk about companies that are publicly traded and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors and we ask that you do not treat us as such.
Transcript:
Daniel Newman: Hey, everyone. Happy Monday. It’s Daniel Newman. Welcome to a special Six Five conversation. It’s a big news day, and if you’ve been following my stream for a while, you know that I’ve been tracking very closely all things AI accelerators, GPUs, what’s going on in the overall AI space. We’re sizing the market. We’re looking at new architectures and everything from next-generation architecture designs to value being created from AI at the customer level. You’ll also know though that you’ve seen me post probably a few times about Groq, a company that I have been advising for several years, and that I am also an investor in, putting that out straight upfront.
And let’s just be candid. As an analyst, an advisor, and consultant, it’s very rare that I join a cap table or an investment realm. It’s got to be something that just absolutely is mind-blowing and awesome. And from the time I met with you and the team over at Groq, I saw so much potential in what you were doing. And of course, there’s been stops, there’s been starts, there’s been iterations, there’s been changes, but you and your team at Groq had a great day today. So all that work you’ve been doing, what’s going on? What news do you have to share with our community today?
Mark Heaps: Well, we’re happy to announce that we have now completed a series D fundraising round for Groq, which means the future looks really, really bright for us. We’ve managed to raise over $600 million. And it’s exciting times because we see where the market is going, we see all the growth and all the demand. You hear about even the incumbents talking about it. And as a startup, you’re wondering, “Hey, are we going to be around?” And thanks to this opportunity, the future looks very bright.
Daniel Newman: So to be clear, you raised 600 million in a D. If I’m not mistaken, in your C it was 300 million at a post-money 1.1 billion roughly?
Mark Heaps: That’s right. So we’ve actually raised $640 million in this round, and then our valuation has more than doubled considerably. So all good signs to how the market is looking at AI and where they see the application shifting and where people are going to start using this infrastructure in their services and in their solutions. So the big part for us is that people that have learned about us have actually understood that we had something pretty radical as far as the performance from our v1 LPU, which is our processor, it’s a language processing unit. But we’ve always had plans in the roadmap of developing new versions of those systems. And then even the next-generation silicon, which we had already announced as a partnership with Samsung, and that was going to get us from our 14-nanometer chip to a four-nanometer chip. So the amazing part is we had only scratched the surface and we’re already blowing people’s minds with this accelerator and the next generation of the chip is going to do so much more. So again, I think it’s just a great time to be at Groq.
Daniel Newman: It sounds like there’s maybe still some dust settling on the exact post money and that’ll be out to be heard. Nobody ever leaves that blank, but a doubling during a time where the macro has been complicated, interest rates are rising, the market hasn’t necessarily seen a ton of value in anything not NVIDIA GPU and you’ve sort of defied that. You’ve defied that by doubling your valuation during this period of time. We’ve seen some of the other startup AI processing companies stall out a bit, struggling to raise money, struggling to show customer wins. Y’all made a pretty big pivot though, Mark, over the last six months-plus that seems to have been somewhat the catalyst. Not to say you weren’t doing well, like I said, I always saw the vision, but you were plodding along and then it seemed like all of a sudden you made a big acquisition, you made a big change of strategy. Talk a little bit about what that was.
Mark Heaps: Yeah, I think Jonathan Ross, our CEO, always had this vision to bring AI to the world. He would tell me even in the earliest days, “I want anyone with a credit card to be able to have access to this level of compute.” And that was his approach to the theme you hear many companies talking about for democratizing AI. But when OpenAI actually started having ChatGPT become present and available to so many people in the world, the world very rapidly got educated on what the potential was from LLMs and GenAI. So that, very luckily for us, was around this time that we also had our technology matured and was available and we were building a cloud solution. And then you saw suddenly a shift where the open source models became very interesting and competitively capable of a lot of this function for services.
And we all know that open source technologies generally eat private for breakfast. And you’ve heard those announcements from Zuckerberg and others about the models for Meta. You’ve seen things from Mistral with their Mixtral models. And we supported all of those and made those available for free to the developer community. And that was the big change was going back. Not quite six months ago, we decided to take our cloud solution, making it accessible through an API, and it actually uses the same endpoints as OpenAI. So anybody that already had an existing application built with ChatGPT could do three simple steps, literally swapping a word three times in their code and it would actually run on Groq using any of the models that we were providing. So we went from having 10 to 20,000 people using our chat LLM interface to suddenly providing a cloud solution, which was developed actually as a part of the acquisition of a company called Definitive.io, Definitive Intelligence with Sunny there as their leader, Sunny Madra.
And they really are experts in this field, and that amplified a lot for us. They knew what the developers wanted, they know how they work, they knew what that trend had been, how people had adopted OpenAI. So they just brought us into another level of service while sticking with Jonathan’s vision to say, “Hey, let’s make it free.” So once that happened, people started messaging. We had this virality moment where they just said, “We can’t believe it’s this fast. We can’t believe it runs the way that it runs.” And a lot of people were skeptical. So what we’ve really been doing over the last several months is every time a new model comes out, we add it, we make it available, and it continues to prove that the performance and the quality is there when it’s powered by Groq. So again, I can’t say enough thank-yous to the people in the developer community who have just been willing to try something different and break away from the incumbents.
Daniel Newman: Yeah, it definitely was a noticeable shift. And after that Definitive deal, Sunny brought a ton of credibility. There were some pretty nice highlight moments on the All-In Pod. We know Chamath is on the cap table and has been involved.
Mark Heaps: Earliest investor, yeah.
Daniel Newman: Yeah, he’s got a very influential platform. But I’ve been watching how you’ve grown virality in the brand, how you’ve really gone out and focused on this open-source community, starting to watch the way the developers talk. Just recently, for the past few days, I was sharing a developer had built a generative AI search using all open source tools running inference on Groq, and he’s just absolutely elated. He was elated by the speed of that inference. And I went on and played with it a bit too. And just seeing how someone can build an all-open source completely open and available tool like that and make it really compelling and really fast is great. Now, I do want to ask a question. It’s big news, $640 million, double your valuation, customers are growing, developers are growing. This is something though that I’m going to ask because you remember televisions, Mark, you’re a brand and a creative guy, remember television resolutions got to the point and every year it was 480p, 720, 1080.
Mark Heaps: That’s right.
Daniel Newman: And it got to a point where I don’t know about you because it was like, “Okay, I get it. It’s cool.” You could make the same argument about iPhones, 13, 14. I’m still on a 13 Pro Max and the reason I haven’t upgraded yet, I don’t know the difference. Why is being able to generate so many tokens so great because I can’t read the text faster than it’s being created? I know this is cool, but for everyone out there that’s asking that question, how does Groq stand for this besides just making it fast? You don’t want to just become the next hypercar. There’s not a big market for that. You need to be a car that can fill the roads, get people to… So what’s the thought process Groq has on being more than just fast and why is fast so important?
Mark Heaps: Well, I think you have to understand as well where are the problems that are starting to arise as the enterprise adopts this technology. It’s great when the consumer-level user is using these LLM chat-like experiences as a replacement for Google, “Hey, I want to ask a question, I get an answer.” And that’s a pretty fundamental way you see a lot of people using chat in an LLM, and that’s great. But when you start talking about, “We want to deploy enterprise-scale applications,” the reason these models keep getting bigger is because they’re trying to improve the quality of the models. Now, we’ve heard people talk about things like using a RAG database, which gets a little technical so that you can have specific information and data relevant to the application. You hear people talk about fine-tuning. You hear people talking about all kinds of things.
But what happens is every time you insert some layer to the infrastructure in the stack of your application, and now the hot topic is everyone talking about mixture of agents, you’ll hear Andrew Ng talking about this, you’ll hear a number of experts talking about it, where you’re not just talking with an LLM, but you’re talking with multiples of an LLM that have different sys prompts, meaning they have different personality types. You have different personas in the way that it reviews the prompt input, which means each one of those ingest tokens generates tokens. And then you typically have some master LLM agent that creates the output as you’re using all these other agents to consider the variables. So you could say one is a really creative agent to write an answer, and you could say another one is a very grammatically specific and very rigid type of agent. And then whatever they both produce, the master agent takes the best of both. All of those tokens being generated are going to slow down your application.
So what you’re really doing is you’re introducing improvements in quality with bottlenecking at scale. So if we’re already, to the point that you made, faster than any human being can read, that means we’ve created a massive margin of opportunity to improve the quality of applications for the enterprise while still delivering at a speed that humans find useful. Now, that’s if you’re doing those human interfacing applications, but then you get into the world of financial trading where they’re dealing with streaming data and that’s no longer about the speed that you’re reading words at, that’s actually the applications taking in that data and being able to answer questions faster than anyone else. And that’s the difference between you getting the jump on the deal or not.
Daniel Newman: Yeah. Well, I think you made a great point though there that there’s the part of the token generation for language of what we’re consuming. And then there’s the part of the ability to actually access all that data and generate because when you get more and more complicated data sets, and of course, we know that there’s a bit of the holy grail, Mark, is the tie-up, it is the conjoined relationship that’s going to exist between data sets, right? The graph, the entire database of all of the company’s data assets, estates, structured, unstructured, prime cloud across multiple clouds, and to be able to train models that can then read all that quickly, whatever technique, whether it’s RAG technique, whether it’s some fine-tuning technique that’s able to access all that and then give a good answer at the fastest pace possible, it does create… So really, in the end, it sounds to me, if I’m going to stick with the car analogy because I like that, it’s like the M Series BMW. You want to create something that’s fast, but high production. You want to create a fast and high production. You’re a motorcycle guy, so a lot of those sport bikes can run pretty darn fast.
Mark Heaps: Well, I think for us, you’re seeing a lot of people in the model-building community experiment. So, for example, we’ve seen Google build these models that have massive sequence context links. And the reason for that, and we’ve even seen in recent papers that when you have a large context length, meaning what can it retain as an input and output at one instance, there’s some studies now showing that improves quality greatly versus, say, let me have a RAG over to the side. So the good thing for us is we don’t build the models. So as we see the models continue to advance, we just need to make sure we have that engine for the M Series that people can really get that performance out of. And for customers that are in the enterprise, they no longer need to fear this idea that, “I’m stuck with this one model and now that’s my software stack for the next 12 to 18 months.” Rather say, “Hey, Groq, we hear that this new model’s coming out and it’s going to be superior in these ways. Will you be able to deploy it?” And of course, we will because we’ve done that since day one. So for us, imagine being able to swap out that M-Class whenever the new version comes out. And that’s really a brilliant opportunity.
Daniel Newman: I’m just going to call you Jonathan Shelby from here on out. Jonathan Shelby. So I got a minute left here with you, Mark, and I really appreciate you taking some time with me. I’m very excited to see this next wave of opportunity and growth for Groq. I’ve been a big proponent of competition in this particular space. I think innovation drives innovation, iron sharpens iron. I totally think having more strong competitive companies, and I think the overall growth of the AI market is palpable despite some of the concerns about when measurable value reaches industries. And I think democratizing open source low-cost available access is going to make that more possible when you’re paying 3,000 for a chip instead of 30,000 for a chip. There is some economics here. I know Jonathan always talked about bringing the cost of compute down to zero. And I also know Jensen likes to say, “The more you buy, the more you save.” So they’re both saying things similarly the same way.
But listen, the last thing I want to talk to you about is there was this pivot, right? You all went from like, “We might be training inference and chip.” Then you went to, “We’re just inferencing chip.” Then you kind of went, “We’re cloud inference and we’re going to make this more of an accessible sandbox for development.” But then it seems to me that there is some really substantial growing demand for your silicon too. It isn’t just going to be, by the way, access to silicon through software. There’s some people that are saying, “Look, we want to use your LPU. We’re going to stand up our own stuff. We’re going to use your software, we’re going to build our own software on it.” Just talk a little bit about how that’s evolving because I think that’s important for your economic story is that you’re going to be able to move volume of chips for people that want to buy accelerators, and then of course, you’re also going to continue to democratize and make affordable access to the sandbox.
Mark Heaps: Yeah, so we call this our ramp to access. So for somebody that wants to build software and not have a major overhead like a lot of the startups and VC-funded projects out there, they can jump on the cloud solution, grab an API key for free, and start building. But then you talk about the hardware side of it and, obviously, there’s going to be instances where people want the hardware on-premise for security reasons and other proprietary reasons. So that’s when you look at the deals that we’ve been doing with Aramco Digital in the Kingdom of Saudi Arabia, you look what we’ve done with Earth Wind & Power out of Norway where they’re talking about building these inference compute centers. And obviously, you need to have some range of how you provide that solution. So we’ve got the cloud at the base end.
You’ve got the ability to have something dedicated for a customer in a data center that we can manage or they can manage, and then you’ve got all the way up to selling them racks that they can put on-prem for themselves. So we’re working all of those deals right now with a large variety of customers. And a big reason that became the potential it has is because the developer community has built applications literally in hours that have proven thousands of times now that this technology can make a difference. So that’s really captured the world’s attention and the question has come back around, “So what can we do with your hardware?” And that is really what I believe inspired a big part of this round. And that’s the first commitment we’ve made with this series D investment is the millions of LPUs that we’re going to be deploying. So those orders have been placed, we’re talking to the foundries, and we’re going to be very, very busy and growing really, really fast over the next 12 months.
Daniel Newman: Yeah. Well, look, you have a lot to be proud of. You and I have been speaking for years. I’ve got probably no less than four or five sit-downs with Jonathan. I know you’ve been a big part of turning Jonathan from the magician in the lab to quite a prolific speaker. And I think as the company continues to grow from billion to billions in valuation and hundreds of thousands of developers and some really big new customers, I know you’ve brought new team members on, you brought Stuart Pann who most recently has been running the foundry business for Intel going to come over here with Groq. So that’s very exciting as well.
But, Mark, congratulations. Thanks so much for making some time. I know you’re off to talk at a bunch of events. The company’s been on and featured everywhere. Hit that subscribe. Check out all our content here on Six Five within The Futurum Group. We’ve covered Groq for years. Check out all my content across social. I’m breaking down the AI space, the chip market every single day here, CNBC, Bloomberg, you name it. But kudos, congratulations. Pass my best on to Jonathan, Adam, and the team. And, Mark, we’ll be seeing you soon. So thanks so much. Talk on, go on, rock on, Groq on. See you later.
Mark Heaps: Thanks so much. We’ll talk to you guys again.
Author Information
Daniel is the CEO of The Futurum Group. Living his life at the intersection of people and technology, Daniel works with the world’s largest technology brands exploring Digital Transformation and how it is influencing the enterprise.
From the leading edge of AI to global technology policy, Daniel makes the connections between business, people and tech that are required for companies to benefit most from their technology investments. Daniel is a top 5 globally ranked industry analyst and his ideas are regularly cited or shared in television appearances by CNBC, Bloomberg, Wall Street Journal and hundreds of other sites around the world.
A 7x Best-Selling Author including his most recent book “Human/Machine.” Daniel is also a Forbes and MarketWatch (Dow Jones) contributor.
An MBA and Former Graduate Adjunct Faculty, Daniel is an Austin Texas transplant after 40 years in Chicago. His speaking takes him around the world each year as he shares his vision of the role technology will play in our future.
