The News: NVIDIA’s acquisition of Run:ai aims to enhance customer efficiency in managing artificial intelligence (AI) computing resources through Kubernetes-based workload management. Run:ai’s platform enables enterprises to optimize their compute infrastructure across cloud, edge, and on-premises environments, supporting diverse AI workloads and enhancing GPU cluster resource utilization. Read more on the NVIDIA website.
GPU Giant Grabs AI Acceleration: NVIDIA’s Acquisition of Run:ai
Analysts Take: Companies considering the use of generative AI (GenAI) applications face myriad challenges. Beyond the issues around how to tailor these applications for a particular business need are the operational challenges of running AI applications in production environments. It is one thing to develop a proof of concept, but something quite different to place them into production.
The tools that Run:ai provide are designed to assist companies in operationalizing AI workloads and help with both fine-tuning models and inferencing. Based on feedback shared from IT consumers and AI practitioners, along with our experiences of running GenAI applications, there are multiple common operational issues:
- Workload management, the ability to monitor and control multiple instances
- Scaling, the ability to distribute workloads across multiple resources
- How to run containerized AI apps within Kubernetes
- Memory management, to help overcome accelerator memory limits
Run:ai’s control plane and cluster engine are designed to help address several of these challenges, including managing workload, scaling, and running containerized instances. Run:ai’s ability to virtualize GPUs is an interesting capability, although it does not itself solve the challenges outlined.
Based on our experiences and interactions with AI practitioners, one of the primary challenges when running AI workloads is the memory constraint of the accelerator card used. This is true for fine-tuning as well as running AI in production (aka inferencing), where a lack of accelerator memory really becomes an issue.
For example, attempting to inference Llama-2-7b, Mixtral-8x7B, or the new Llama3 models on a NVIDIA RTX-4080 card would likely produce an out of memory error. These issues can be mitigated through techniques such as quantization, which reduces the size (and accuracy) of the models. However, you can only quantize a model so far, and to overcome these limitations, you must either use an accelerator with more memory or use additional accelerators and split up the workload.
This problem is not limited to consumer accelerators, as even the Intel Gaudi2 with 96 GB and the NVIDIA H-100 with 80 GB of memory can result in errors when inferencing larger models such as Llama-3-70b or similar without quantization or scaling out the workload across multiple cards. Scaling workloads can be challenging, and this is where NVIDIA’s acquisition of Run:ai may help IT enterprises operationalize AI workloads.
Using Run:ai will not solve every operational issue, and challenges remain for enterprises looking to implement real AI-powered business solutions; however, Run:ai may help companies overcome common challenges.
Future Outlook
Companies venturing into GenAI applications encounter numerous operational hurdles beyond mere customization for specific business needs. Transitioning from proof of concept to production entails grappling with workload management, scaling, containerized deployment within Kubernetes, and memory constraints inherent in accelerator cards.
Run:ai’s tools aim to alleviate some of these challenges by facilitating workload management, scaling, and containerized deployment, though they do not entirely resolve memory limitations. Nevertheless, leveraging Run:ai could aid enterprises in operationalizing AI workloads and navigating common hurdles, even as challenges persist in implementing robust AI-powered business solutions.
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Other Insights from The Futurum Group:
Oracle and NVIDIA Boost Sovereign AI Globally
Pure Storage and NVIDIA Announce New Reference Architectures for AI
Image Credit: NVIDIA
Author Information
With over 25 years of experience, Paul has a proven track record in implementing effective go-to-market strategies, including the identification of new market channels, the growth and cultivation of partner ecosystems, and the successful execution of strategic plans resulting in positive business outcomes for his clients.
Russ brings over 25 years of diverse experience in the IT industry to his role at The Futurum Group. As a partner at Evaluator Group, he built the highly successful lab practice, including IOmark benchmarking.
Prior to Evaluator Group he worked as a Technology Evangelist and Storage Marketing Manager at Sun Microsystems. He was previously a technologist at Solbourne Computers in their test department and later moved to Fujitsu Computer Products. He started his tenure at Fujitsu as an engineer and later transitioned into IT administration and management.
Russ possesses a unique perspective on the industry through his experience as both a product marketing and IT consumer.
A Colorado native, Russ holds a Bachelor of Science in Applied Math and Computer Science from University of Colorado, Boulder, as well as a Master of Business Administration in International Business and Information Technology from University of Colorado, Denver.
