The Six Five team discusses Groq’s US Army Entanglement Report.
If you are interested in watching the full episode you can check it out here.
Disclaimer: The Six Five Webcast is for information and entertainment purposes only. Over the course of this webcast, we may talk about companies that are publicly traded and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors and we do not ask that you treat us as such.
Transcript:
Daniel Newman: Let’s talk about Groq. So you and I are both advisors and early investors. I want to point that out in the company and Groq’s a really exciting one. Right now with the semiconductor industry under fire, you’re seeing a lot of accelerator technology companies kind of wondering what their long term trajectory is. Pat, I feel like we bet on a good one and Groq, under the company’s leadership announced over the last couple of weeks, basically a partnership in depth between them and the US Army and Entanglement, which is a company that partners with the army to work on some strategic AI projects. And they did a big validation and they were working on cyber security using what they call a Groq node server running Entanglement software to effectively identify security anomalies.
And the outcome, Pat, long and short, was a hundred, I’m sorry, was three orders of magnitude faster with lower numbers of false positives. Now this is pretty techy stuff, but just to get to the bottom line, the work in the past, and again we talk about inferences, we talk about things like tops, these are big numbers and something trillions of ops, stuff like that but basically in order to, with the amount of data moving around the web, you have to have the ability to look at a lot of concurrent data at the same time to try to identify some type of anomaly that could bring a security risk and if the military isn’t the key example of where this becomes really important, then I don’t know where it is because we need to identify things as early as possible and we need the best technology to do that.
Well basically past validation reports had this type of anomaly detection, it’s somewhere around like 120,000 inferences per second. Now you talked about three is magnitude. The Groq AI approach with Entanglement got that up to, in the test, and the validation of 72 million inferences per second. So from 120,000 to 72 million and they believe that with this current technology on their workloads, that could be pushed all the way to a full 120 million inferences per second. I mean long and short Pat, really important when you think about the types of threats the army, other military branches and potentially the US government are trying to deal with when it comes to cybersecurity. So the army basically came out and it’s not an easy thing, I think it’s important we reiterate that. It’s not an easy thing to get any US Army or military or even the government to come out and kind of tout a single technology.
So it was a huge moment of validation for what is still a mid to late stage startup in Groq, but effectively just the kind of simple math here, Pat, a thousand times cybersecurity performance running their system over the past technology that was being used and they’ve tried everything, by the way, long and short, Quantum, they tried that. They tried to use a software hardware bridge and effectively after all these different things they tried, they ended up with Groq. They tested with Groq and got this amazingly remarkable performance. So Pat, I think we picked a good one. This is a good moment. It’s far from making it a hugely profitable company, but it’s that kind of moment where you start turning corner. If it’s good enough for the army, who else is it going to be good enough for? And I think this is a good indicator of their future.
Patrick Moorhead: Yeah, this is one of the best third party of what Groq has been saying for a long time. I did a really long interview with Groq CEO Jonathan Ross. I really wanted to get underneath the why, like how did you do this? And he did tell a similar story that talked about what CPU, GPUs and FPGAs are good at and what he wanted to do was come up with a much lower latency accelerator that’s essentially an ASIC that solved the challenge of batch size one. And essentially when we see a lot of these benchmarks out there, they use big batches. So latency isn’t necessarily in latency, just a fancy way to say responsiveness. And how do you get great responsiveness with batch size one that are a lot harder to do on inference platforms based on a GPU. And for everything I’ve seen, the Groq architecture doesn’t experience any latency at batch size one and it’s single threaded, single core architecture really has consistent performance and latency across any batch size.
It’s funny, what the US Army said was actually bigger than what Groq claims, who says their TSP is 2.5 times faster than any GPU-based platform at large batch and 18 times faster than GPUs at batch size one. So I got to love it when the manufacturer, Groq, is more conservative than a customer who actually ran these and that just gives me more trust in the company and what the company says, because it hasn’t exactly jumped into the ML perf game with all limbs here, people like Nvidia has. And we just saw one from Intel Habana. I think these customer stories are actually more influential to me than the benchmarks.
Daniel Newman: And there you have it. I think Pat, that’s a good assessment by the way. I like that you mentioned some of the technical capabilities and how you got there with Jonathan. He’s an incredibly smart guy. Did he show you his egg timers?
Patrick Moorhead: No he hasn’t. No.
Daniel Newman: Okay. I was out there. I actually went out to their offices and I had a conversation with him and it was just absolutely fascinating although I only understood half of what he said, and I consider myself a pretty smart guy so he’s a brilliant guy and they’re building something, I think, that’s going to be really special out there Pat.
Author Information
Daniel is the CEO of The Futurum Group. Living his life at the intersection of people and technology, Daniel works with the world’s largest technology brands exploring Digital Transformation and how it is influencing the enterprise.
From the leading edge of AI to global technology policy, Daniel makes the connections between business, people and tech that are required for companies to benefit most from their technology investments. Daniel is a top 5 globally ranked industry analyst and his ideas are regularly cited or shared in television appearances by CNBC, Bloomberg, Wall Street Journal and hundreds of other sites around the world.
A 7x Best-Selling Author including his most recent book “Human/Machine.” Daniel is also a Forbes and MarketWatch (Dow Jones) contributor.
An MBA and Former Graduate Adjunct Faculty, Daniel is an Austin Texas transplant after 40 years in Chicago. His speaking takes him around the world each year as he shares his vision of the role technology will play in our future.
