Groq Meta LLAMA-2 70B Parameters 100 tps Milestone

Groq Meta LLAMA-2 70B Param 100 tps Milestone

The Six Five team discusses Groq’s milestone of running Llama-2 70B at more than 100 tokens per second.

If you are interested in watching the full episode you can check it out here.

Disclaimer: The Six Five Webcast is for information and entertainment purposes only. Over the course of this webcast, we may talk about companies that are publicly traded and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors and we ask that you do not treat us as such.

Transcript:

Daniel Newman: Groq made a pretty big announcement about a hundred tokens per user per second. Pat, what does that mean?

Patrick Moorhead: Yeah, so good question. So first of all, Groq is a company that was founded by the folks that did the Google TPU. So smart cookies. And in my vernacular, they’re creating an ASIC to tackle first inference and then training. As we talked about many times on this show, an ASIC is more efficient than a GPU at doing certain things. And then the challenge is putting a programmatic layer on top of the ASIC to make it programmable. And then there’s Llama 2. So Llama 2 is an open source model that came out of Meta that everybody but trillion-dollar companies can take advantage of for free. And essentially it’s all the rage, right? Open models, right? Because we don’t want one company to have their model.

And what do I mean by closed models, right? So OpenAI and ChatGPT is a closed model. Bard is a closed system as well. So now, you have in the enterprise world at least everybody’s saying, “Hey, it’s about a combination of proprietary and open models distributed through somebody like a hugging face.” And then the 70 billion parameter model where they were literally according to them. And I can’t find any data that says this is not, it’s the fastest performance on Llama 2 70 billion parameter at over a hundred tokens per second per user. And the reason tokens are important as tokens determine the amount of data that can go into the prompt or they can go into the grounding.

So this has a lot to do with the pricing as well. So the cool part is that the cost is just extraordinarily lower to do this. And Dan, you hit this on the NVIDIA piece. Groq says that on a workload like this you get three X lower total cost of ownership from the inception, which is really great value, right? Those are comparisons using an 80-node NVIDIA A100 SuperPOD is $27 million, and H100 SuperPOD is $39 million. And a Groq 80 node system is $18 million. So again, competition is good. Dan, that’s a theme on our show. We say it every day. Competition matters. And one final thing, current silicon is 14 nanometer. Imagine when they get to four nanometer or five nanometer, performance and power should be amazing.

Daniel Newman: Absolutely. So I’m going to keep running. I’ll just say in the press release I did comment availability, Pat. I mean, you can actually buy these things. I just want to point that out. These are actually available which and surprise people wouldn’t want to capitalize on that.

Author Information

Daniel is the CEO of The Futurum Group. Living his life at the intersection of people and technology, Daniel works with the world’s largest technology brands exploring Digital Transformation and how it is influencing the enterprise.

From the leading edge of AI to global technology policy, Daniel makes the connections between business, people and tech that are required for companies to benefit most from their technology investments. Daniel is a top 5 globally ranked industry analyst and his ideas are regularly cited or shared in television appearances by CNBC, Bloomberg, Wall Street Journal and hundreds of other sites around the world.

A 7x Best-Selling Author including his most recent book “Human/Machine.” Daniel is also a Forbes and MarketWatch (Dow Jones) contributor.

An MBA and Former Graduate Adjunct Faculty, Daniel is an Austin Texas transplant after 40 years in Chicago. His speaking takes him around the world each year as he shares his vision of the role technology will play in our future.

Related Insights
OWE-Bench Exposes AI Agents' Long-Horizon Ceiling
November 17, 2026

OWE-Bench Exposes AI Agents’ Long-Horizon Ceiling

Mercor's OpenWorldEngineer benchmark exposes fundamental limitations in AI agents' ability to handle long-horizon engineering tasks, with Claude Fable 5.2 leading at 54.6%—signaling enterprise adoption challenges....
GE Vernova's SF₆-Free Switchgear Targets AI Grid Bottleneck
November 17, 2026

GE Vernova’s SF₆-Free Switchgear Targets AI Grid Bottleneck

GE Vernova's HYpact-145g tackles grid interconnection constraints affecting 16.6% of AI operators. Its SF₆-free switchgear reduces substation footprint by 50%, enabling faster deployment in space-constrained environments....
EY and Stuut Bet on Agentic AI to Fix Enterprise Cash Flow
November 17, 2026

EY and Stuut Bet on Agentic AI to Fix Enterprise Cash Flow

EY US and Stuut allied to combine finance transformation with agentic AI, helping enterprises reduce Days Sales Outstanding and unlock working capital. Stuut's platform has moved $3B+ and cut DSO...
Schneider Electric and Arup: Solving AI's Power Problem Together
November 17, 2026

Schneider Electric and Arup: Solving AI’s Power Problem Together

Schneider Electric and Arup announce a co-development partnership to solve AI's power problem through advanced energy management and microgrid expansion, as global data center demand is projected to double by...
UK Enterprises Are Done Experimenting With AI
November 17, 2026

UK Enterprises Are Done Experimenting With AI

Creatio's latest report reveals 83% of UK enterprises have progressed beyond AI pilots into active agentic AI deployment. With 86% expecting AI agents to reshape workforce capacity, governance and oversight...
Tenable One Adversary View: Fixing What Attackers Would Exploit First
November 17, 2026

Tenable One Adversary View: Fixing What Attackers Would Exploit First

Tenable One Adversary View, now generally available, uses Claude Mythos' adversarial reasoning to rank vulnerabilities by attack-path impact rather than severity alone, helping security teams address what attackers would exploit...

Book a Demo

Welcome

The vision behind everything in Futurum’s Custom Research practice is this: research should show you what is happening, what comes next, and what to do about it. It should be personal to each audience, easy for people to grasp, and structured so LLMs can reason over it accurately. And it should be fast and turnkey; you want answers now, not another project to carry for quarters.

Whether you are defining business, channel, or go-to-market strategy; evaluating vendors or justifying ROI; or commissioning research to fill an emerging market need, we have your back, with a program that answers your questions with the objectivity and credibility to drive real decisions.

To do it, we bring unmatched data to bear: Futurum research, surveys, and market projections; validated market feeds; ETR’s 15 years of insight from 10,000 technology decision-makers; G2’s buyer and user data; and what our analysts hear every day. Add leading primary collection, from AI-moderated voice interviews to surveys and analyst-led interviews, all turnkey, and every project comes out credible, nuanced, and actionable.

And we don’t just drop the results in your lap. For internal work, we provide analyst-led sessions, interactive dashboards, and a range of formats. For market-facing work, Futurum delivers turnkey activation and amplification that actually gets seen, by people and by LLMs, through our media and share of voice. This is research that moves decisions and markets.

We will meet you wherever you are, from a fast-turn brief to a multi-year program, and shape the work to your goals, timeline, and budget. The right program for your moment.

If any of this is useful, I would love to talk.

Benjamin Brown, VP Custom Research, Futurum Research

Benjamin Brown

VP, Custom Research · The Futurum Group

Newsletter Sign-up Form

Get important insights straight to your inbox, receive first looks at eBooks, exclusive event invitations, custom content, and more. We promise not to spam you or sell your name to anyone. You can always unsubscribe at any time.

All fields are required






Thank you, we received your request, a member of our team will be in contact with you.