Introduction: Generative AI is widely considered the fastest-moving technology innovation in history. It has captured the imagination of consumers and enterprises across the globe, spawning incredible innovation and along with it a mutating market ecosystem. Generative AI has also caused a copious amount of FOMO, missteps, and false starts. These are the classic signals of technology disruption – lots of innovation, but also lots of mistakes. It is a rumpus room with a lot of “kids” going wild. The rumpus room needs adults. Guidance through the generative AI minefield will come from thoughtful organizations who do not panic, who understand the fundamentals of AI, and who manage risk.
Our picks for this week’s Adults in the Generative AI Rumpus Room are Arthur, YouTube and AI2
Arthur Bench: A Tool for Evaluating LLMs
The News: On August 17, AI model monitoring startup Arthur announced it has introduced Arthur Bench, an open-source evaluation tool for comparing large language models (LLMs), prompts, and hyperparameters for generative text models.
Some of the key features of Arthur Bench:
- Model selection and validation: Helps compare different LLM options available using a consistent metric so businesses can determine the best fit for their application.
- Translation of academic benchmarks: Companies want to evaluate LLMs using standard academic benchmarks like fairness or bias, but have trouble translating the latest research into real-world scenarios. Bench helps companies test and compare the performance of different models quantitatively so that they are using a set of standard metrics to evaluate them. Companies can configure customized benchmarks that they care about, enabling them to focus on what matters most to their specific business.
Alongside Arthur Bench, the company launched the Generative Assessment Project (GAP), a research initiative ranking the strengths and weaknesses of LLMs from OpenAI, Anthropic, Meta.
Read the full announcement for Arthur Bench here.
Adults because… In a nascent market with so many unknown and unproven LLMs, enterprises need to take a pragmatic approach in evaluating their options. Up to this point, that process would require a lot of organic legwork and a gut-feel evaluation. Arthur Bench gives enterprises an opportunity to compare LLM performance and features with a defined criteria (though to be fair, we do not know what that is or whether or not it makes sense).
YouTube Enlists UMG Artists to Tinker in YouTube Music AI Incubator
The News: On August 21, YouTube announced they are launching a new initiative called the YouTube Music AI Incubator. The blog post by YouTube CEO Neal Mohan, the company said they have enlisted a range of Universal Music Group’s artists to “…help gather insights on generative AI experiments and research that are being developed at YouTube.”
Mohan said in partnership with artists, YouTube intended “to develop an AI framework to help us towards our common goals. These three fundamental Ai principles serve to enhance music’s creative expression while also protecting music artists and the integrity of their work.”
The principles are:
- AI is here, and we will embrace it responsibly together with our music partners.
- AI is ushering in a new age of creative expression, but it must include appropriate protections and unlock opportunities for music partners who decide to participate.
- We’ve built an industry-leading trust and safety organization and content policies. We will scale those to meet the challenges of AI.
Read the full blog post on YouTube Music AI by CEO Neal Mohan here.
Adults because… Generative AI is both a technology of potential and one of threat. Perhaps that is nowhere more evident than in the creation of and protections required for media content. YouTube and parent Alphabet/Google have a lot at stake in this area particularly when it comes to music content, so the fact that the company has initiated a project to address generative AI and music is a positive step. Of course, YouTube is not doing this out of the goodness of their hearts, but if they are able to hammer out frameworks where artists’ work is protected, or they are compensated fairly for their work being used in AI training, or artists using generative AI to create new content have a legitimate path to do so, it could serve as the foundation for other artists and platforms to improve upon.
AI2 Debuts Open Dataset for AI Training
The News: On August 18, the Allen Institute for AI (AI2) announced the availability of Dolma, a dataset of 3 trillion tokens. It is the largest open dataset to date. Dolma is the dataset AI2’s planned open source LLM, OLMo, will be based on. Nearly all datasets on which current LLMs are trained are private.
Read the details of DOLMA on the AI2 blog.
Adults because… Most LLMs have been built on datasets that are private. The data is typically scraped, without permission, from publicly-available data on the web. The major challenges of LLM outputs include bias, toxicity, inaccuracy, and hallucination. One way to address these issues is for those who use the LLMs to be able to trace these issues back to the data source. Open datasets provide that opportunity.
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Other insights from The Futurum Group:
Meta Introduces SeamlessM4T Model in a Step Toward a Universal Translator
YouTube Enlists UMG Artists to Tinker in YouTube Music AI Incubator
Adults in the Generative AI Rumpus Room Cohere, IBM, Frontier Model Forum
Adults in the Generative AI Rumpus Room: Google, DynamoFL, and AWS
Author Information
Based in Tampa, Florida, Mark is a veteran market research analyst with 25 years of experience interpreting technology business and holds a Bachelor of Science from the University of Florida.
