Introduction: Generative AI is widely considered the fastest-moving technology innovation in history. It has captured the imagination of consumers and enterprises across the globe, spawning incredible innovation and along with it a mutating market ecosystem. Generative AI has also caused a copious amount of news and hype. To avoid AI FOMO and find the right path, the wise will pay attention to trends and not be distracted by every announcement and news bite.

AI Compute Is Shifting to Meet Demand for Cheaper, Better Outcomes

The News: Recent announcements from semiconductor companies around AI compute include the following.

• Groq’s language processing unit (LPU) breaks LLM performance record. On August 8, startup AI chipset provider Groq announced it now runs LLM Llama-2 at more than 100 tokens per second, per user on a Groq LPU. Tokenization is a process LLMs use to break text into smaller, manageable units. It allows LLMs to process text more efficiently by reducing memory requirements and compute complexity. The more tokens per second per user an LLM can process, the faster the LLM will load results for users and the less AI compute the application will require.
• Qualcomm Cloud AI 100 chip leads in power efficiency testing. As AI and ML workloads grow in size and complexity, they demand more computing resources and energy consumption. This poses a challenge for both providers and end users who want to deliver and access high-quality services at a reasonable cost. Therefore, it is essential to perfect the performance and efficiency of the solutions that support these workloads. Qualcomm Technologies offers a wide range of power-efficient AI accelerators to meet the performance/TCO requirements.
• AMD introduces chip designed for specifically for generative AI workloads. On June 13, AMD introduced the AMD Instinct MI300X chip to provide compute and memory efficiency needed for LLM training and inference. AMD says its graphics processing unit (GPU) is highly efficient and that AI workloads using them require less GPUs than competitors.
• Kneron launches latest version of its neural processing unit (NPU). According to the company, the chip tackles one of the largest bottlenecks to widespread AI adoption: the high costs driven by prevailing energy-inefficient hardware. The KL730 yields a 3 to 4 times leap in energy efficiency compared to previous Kneron models and is 150% to 200% more energy efficient than major industry peers.

Analyst Take: The current state of AI compute is a prime example of the perfectly-timed workaround. GPUs were designed for computer graphics and image processing, primarily for high-end computer gaming graphics cards. They were then found to be useful in multitasking and running programs in parallel. Consequently, they can execute more mathematical calculations with greater efficiency than central processing units (CPUs), which make them a good solution for AI compute, particularly for AI training. However, AI workloads are massive and therefore, expensive. Generative AI workloads are even bigger than legacy AI workloads and to make generative AI applications viable, compute cost has to come down. Semiconductors have long product cycles, but fortunately, there has been significant innovation and progress in developing purpose-built AI chips. Chip wars will heat up and the winners will be enterprises that are willing to experiment with AI compute at competitive prices.

AI Models/LLMs Are Mutating and One Size Does Not Fit All

The News: AI models/LLMs are mutating. How? They are getting smaller and more specialized:

• Developer creates a nano version of already-small Llama-2, demonstrates ability to run powerful LLM on laptop. Meta’s Llama-2 has quickly inspired LLM-powered applications at the edge and on constrained devices.
• SK Telecom taps Anthropic to build telco AI use case-specific LLM. On August 13, SK Telecom told the media it will invest $100 million in startup LLM Anthropic. The company said this in the announcement: “Anthropic will work with SKT to fine-tune Claude to telco use cases, including industry specific customer service, marketing, sales, and interactive consumer applications. By customizing the model to the telco industry, telcos will benefit from increased performance relative to the use of more general models.”
• Sakana AI leverages biomimicry to design AI models that are flexible and reactive. Ex-Google scientist launches an AI model concept that will develop multiple smaller AI models which collaborate to deliver complex results. The swarm-based approach is an attempt to address how inflexible and monolithic new AI models are.
• Startup Cohere is valued at more than $2 billion, focused on customization and fine-tuning. Customers get full control over data security, host in private or public cloud or Cohere’s managed cloud. Custom model training.

Analyst take: First-generation LLMs are trained on massive amounts of public data. Numerous challenges are inherent to that approach – the quality of the data causes bias, inaccuracy, hallucinations, and more. Next-generation LLMs are taking a more measured approach to address these issues – enabling companies to train AI models on their own data sets, building industry-specific models, and creating smaller models that can be impactful but less expensive. AI model development will continue to evolve. Savvy enterprises will look for multiple options and design their architectures for better ways to plug and play.

Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.

Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.

Other insights from The Futurum Group:

Cohere Launches Coral, a New AI-Powered Knowledge Assistant

Generative AI Investment Accelerating: $1.3 Billion for LLM Inflection

Generative AI War? ChatGPT Rival Anthropic Gains Allies, Investors

Author Information

Mark Beccue

Mark comes to The Futurum Group from Omdia’s Artificial Intelligence practice, where his focus was on natural language and AI use cases.

Previously, Mark worked as a consultant and analyst providing custom and syndicated qualitative market analysis with an emphasis on mobile technology and identifying trends and opportunities for companies like Syniverse and ABI Research. He has been cited by international media outlets including CNBC, The Wall Street Journal, Bloomberg Businessweek, and CNET. Based in Tampa, Florida, Mark is a veteran market research analyst with 25 years of experience interpreting technology business and holds a Bachelor of Science from the University of Florida.