On this episode of The AI Moment, we explore why Qualcomm could be the key to on-device AI and how Microsoft, which has developed best practices-level AI safety policies and a separately an insightful blueprint for approaching enterprise AI, continues to show they are an Adult in the AI Rumpus Room
The discussion covers:
- Adults in the AI Rumpus Room: In two separate initiatives, Microsoft continues to show their leadership in the responsible use of AI. Microsoft has developed best practices-level AI safety policies and a separately an insightful blueprint for approaching enterprise AI. Download Microsoft’s Building a Foundation for AI Success: A Leader’s Guide
- A company we like doing AI: Qualcomm has just flexed some significant compute muscle and could be poised to be a key player in making on-device AI a massive success.
Watch the video below, and be sure to subscribe to our YouTube channel, so you never miss an episode.
Listen to the audio here:
Or grab the audio on your favorite podcast platform below:
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this webcast.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Transcript:
Mark Beccue: Hello, everybody. I’m Mark Beccue, Research Director for AI at The Futurum Group, and welcome to The AI Moment, our weekly podcast that explores the latest developments in enterprise AI. Now, unless you’ve been hiding under a rock, you understand that the pace and change of innovation in AI has caught everybody by surprise since October of last year when ChatGPT was introduced and kickstarted the degenerative AI time. That’s why we call this The AI Moment. But in this podcast, we try to distill the mountain of information, separate the real from the hype, and provide you with some shorthanded analysis about where the AI market is about to go. In each of our episodes, we’re going to dive deeply into the latest trends in technologies that are shaping the AI landscape, and that’ll be everything from latest advancements in AI technology, parsing the mutated vendor landscape, including big announcements. And we’ll talk sometimes about regulations, ethics, risk management, and more. There’s a lot. Each of our shows is about 30 minutes, and typically there’s three or four segments. The four go like this. Guest spotlight, where we talk to a guest, typically from a vendor company, about how they’re operationalizing AI for their customers. Segment two is key trends in generative AI, and I call that trends instead of fads, right? So we’re going to look at trends and try and map it that way instead of what’s just happening at the second. The third big one is adults in the Adults In The Generative AI Rumpus Room. As you can imagine, the generative AI moment has produced a lot of chaos and disruption and some impatience in the marketplace. Some organizations have been calm and thoughtful leaders through the midst of this, so we like to call them the Adults In The Generative AI Rumpus Room. We like to highlight their work. Finally, the last segment we typically might do is called the company we like doing AI, which is self-explanatory.
So here we are. This is episode four. Today, we’re going to use two segments. We’re going to talk through two segments: Adults In The Generative AI Rumpus Room, and I have an unusual thought through that this week. It’s a double shot from Microsoft, who has two things going on that I felt qualified them for the adult heading this week. And second, we’re going to talk about companies we like doing AI. Had a long time with Qualcomm two weeks ago and got to hear how they’re thinking. A mobile chip giant is flexing some compute muscle and could be poised to be a key player in making on-device AI happen. So that’s a challenge, but we’re going to look at that a little bit. Okay, so let’s go right to it and go into the rumpus room. Like I said, this is a double kudos this week to Microsoft. They did two things. I’m going to cover the short one first. They published a white paper this week, which I’ll put in our podcast notes or a link for you, and it’s a white paper called Building a Foundation for AI Success: A Leader’s Guide. I thought this was really interesting and a good idea and really reflects very much the philosophies that we think of when we’re trying to advise enterprises about operationalizing AI. It covered everything that we were thinking of, and I like the way it’s structured. I think it’s a really, really useful piece that almost any company that’s looking at operationalizing AI should look through. And they stack it. They call it the five pillars. I’m just going to name those for you real quick and what they talk about, but it’s very important that the first thing they say is business strategy. We always say this. It’s like, don’t go get AI for the sake of getting AI. Think about a problem that you have that you’re trying to solve. That’s the most important thing. So Microsoft does that too. They say the first pillar is business strategy. They say clearly defined and prioritized business objectives, use cases and measurement of AI value.
And I’m going to reflect on that for a second and just say for us, sometimes when there’s some disruptions like this, such huge disruptions that we have like AI, companies will lose their business discipline and they don’t think about those steps they take when they go through pragmatic business use cases, why they’re doing something, cost analysis, cost benefit analysis, all of those good things. That always has to take place. AI is no different, and you need to be thinking about going through that process. It’s the first thing you do when you’re looking at AI. So Microsoft sets it up nicely to say that’s the first thing. The second thing they say is called technology strategy, and I’m going to read their little piece. It says, “An AI ready application and data platform architecture align parameters for build versus buy decisions and plans where to host data and applications to optimize outcomes.” So that’s the other part, right? When you go through that business case, it leads you to say, as a business you’d say, “How do we approach this AI? Are we going to build, buy, or partner?” Very important, and it depends on what the resources you have in that company and it depends on the strengths of your company, what you’d rather not focus on versus focusing on. I think there’s really this emphasis right now on build it yourself and that every company has to build their own AI. I don’t think that’s true, but it’s interesting that Microsoft notes this and says this is the next part you look at. The third part they call AI strategy and experience. This would be they call it a systematic customer-centric approach to AI that includes applying the right model to the right use case and the experience in building, testing, and realizing AI value across multiple business units, use cases, and dimensions. So this idea I think is saying, do we have a centralized how we go about the market? Are we thinking about just one thing that we want to apply AI to, or is there multiple things we want to apply AI to?
If there’s multiple things, this really comes into play because you’re thinking about whether you’re maybe optimizing operations and that might be in one department versus you’re looking at top line revenue growth where you’re trying to boost sales or build a product that’s better. All those kinds of things come into place. So that was the third piece. The last two are very important too. We said this, I would say this, their fourth pillar they call organization culture. We talk about this all the time.I was talking about this yesterday at a conference where I was moderating a panel and I said, “The AI Moment is about technology, but it’s really a lot more about people and processes.” What’s your organization look like and how’s the culture for that organization? Is it accepting innovation? Is it one that you can fit AI into easily, or do you have a culture issue? And so they were talking about that a bit. I’m going to skip on and say the last piece and the fifth part of their pillars is AI governance. Very important. In this case, I’m reading their piece, implementation of processes, controls, and accountability structures to govern data privacy, security, and responsible use of AI. Can’t say this enough, a company can’t do AI without understanding and having a risk management framework in place. Because there are so many risks to running AI, to operationalizing AI, you have to understand what all that means. I think they make a very good point of, are you thinking about privacy concerns? Are you thinking about misinformation and disinformation and hallucinations and bias and all those pieces that could affect or impact customers that you have, communities that you’re working with? Any of those kinds of things have to be figured out once you get into this. So I thought this was really cool. They get a kudos for publishing that. We’ll put it in our notes here so that you can see it. But kudo number one to Adults In The Generative AI Rumpus Room Microsoft with that piece.
The second thing they did this past week, it was very interesting, was they published… So you might have remember this past week, there was the AI safety council meeting in the UK. The UK had a big conference about AI safety, and Microsoft published an update to their AI safety policies. So they did this on the 26th in response to the… The UK government actually asked them to do this, so I thought it was kind of interesting. What I think was really interesting about this, I’m going to give you a couple pieces about what they said in the piece and why this was important. So it was very detailed, really cool. Point number one, they went through what they had done in these nine areas of AI practice. I noted that Microsoft was mentioning that they collaborated very closely with OpenAI on responsible capability scaling. Call it that. I’m going to quote what it said here. It says, “When it comes to the frontier model deployment, which are the OpenAI models, Microsoft and OpenAI have together defined capability thresholds that act as a trigger to review models in advance of their first release for downstream deployment. The scope of our review through our joint Microsoft-OpenAI Deployment Safety Board includes model capability discovery.” Good point. They go on and they say that the two companies have “prepare detailed artifacts for the joint DSB to review artifacts, record the process by which our organizations have mapped, measured, and managed risks, including though through the use of adversarial testing and third party evaluations as appropriate.” It went on and said that they check these models independently of OpenAI as it applies to Microsoft’s use of those models.
Quote again, “At Microsoft, we also independently manage a subsequent safety review process. We evaluate model capability as deployed in a product where additional safety mitigations can be implemented and measured for impact to check for the effectiveness and appropriate mitigations prior to release.”
All that said, again, I roll up to say that that OpenAI has really opened the floodgates for generative AI. Very smart people, massive model. As we’ve talked about on this show before, LLMs and some foundation models have flaws. There’s issues with them that have to be mitigated. And I think that it was really important that Microsoft was pointing out the detail of how they are working with OpenAI to make sure those models, what they’re doing with them is responsible. So I thought that was interesting and a really good point. The second point out of the policy that I picked out was they built new policies for what they call model evaluations and red teaming. And what that meant was they developed these, tt’s internal practices and guidance for an AI red team, which is an expert group that is independent of Microsoft product building teams. So AI red teams are responsible to map, measure, and manage the potential harms and the misuse of AI systems, like they talked about before. Some of this work includes you simulate real world attacks. They use exercising techniques that a threat actor might use. It includes practices to map risks outside of traditional security, and those are like benign usage scenarios. Something like prompt injection attacks, that’s a big deal. That’s where content is submitted to an LLM by the user or by a third party, and it results in unintended actions. That’s a prompt injection attack. They look at content harms, so that’s where malicious or benign usage of the system results in harmful or inappropriate AI generated content. And they do privacy harms, same kinds of things. So all these developments led Microsoft to update that internal practice guidance for their security development lifecycle threat modeling requirement, whatever that is.
So they’re building these red teams and it’s very detailed. So I thought that was really interesting as well. It’s longer than that. There’s lots more detail to it, but I wanted to give you a quick look here what I think that means. I thought that was really an adult move to do these things. They get kudos for that. And I think mostly this is because of that idea that they’re really doing their best to safeguard how they’re using partner OpenAI. And I would just go back and say that a lot of folks thought, and I was one of them, that it’s kind of risky for Microsoft to partner with OpenAI. I mean, they’re a startup. They were built to further AI research. They didn’t have a track record in enterprise customers or enterprise grade solutions. So it’s really evident to me that Microsoft’s AI Safety Policy update was showing that they have really thought through the guardrails for leveraging OpenAI. It’s groundbreaking IP. It really is. And they have much to risk. Microsoft has much to risk if they were to misstep here because of these great unknowns. We talked about that. So I think that it’s really important and it’s significant that if Microsoft feels comfortable with what they’re doing, that maybe we’re on the right path here in how they’re leading forward with how these foundation models get used. So kudos to Microsoft twice this week. They’re the adult in the AI rumpus room. I have one more.
So we’re going to move on to companies we like and we’re going to talk about Qualcomm today. So two weeks ago, Qualcomm held their Snapdragon Summit and went out. We were talking about they introduced some new SoCs, chip sets for both PCs and smartphones. I’m going to tell you a little bit about those. One was called Snapdragon X Elite. The other’s called Snapdragon 8 generation 3. So Snapdragon X Elite is built for laptops, and there’s a lot in this thing. They reported, this is Qualcomm’s words, that the CPU in this is two times faster versus their competition in CPU and that’s a one-third of the power. It’s capable of running these generative AI models. Lots of times we talk about how big these are. It’s 13 billion parameters on-device at four to five times faster than competitors, and that’s at a speed of 30 tokens per second. They also said that this Snapdragon X Elite is expected to come out from their partners, their leading OEM partners, by mid 2024. Set that side for a second. We’ll talk about the mobile chip set is called Snapdragon 8 Gen 3. They call it their first mobile platform meticulously designed with generative AI in mind. You can say what you will about that. I’ll get to that in a minute. And what that did, they did another test on running generative AI models there. This one runs 10 billion parameter AI models on-device at 20 tokens per second. So that’s not much different than what they… Pretty good results for both of these chips. They mentioned that these flagship Android devices powered by that chip are expected to be available in the next few weeks.
So a couple other things that came out of the summit. We had a lot of time to talk to executives there about things, and I wanted to note a few things that I thought were really important about what I think Qualcomm is showing some leadership and how they do AI. One was they put a graphic up metric and said that they have an installed base of devices with Snapdragon chips that is over a little over 3 billion-devices. So we’re going to come back to that in a second, but I want you to understand that I think that’s really relevant to why on-device is important. That’s a lot of devices. That’s not nearly all the PCs and smartphones that are out there, but it’s a pretty big chunk. Second was that Qualcomm has gathered a pretty comprehensive range of partners for looking at on-device AI and that includes Microsoft and Meta, Google and Baidu, OpenAI, Bloom Stability AI, and then a few that we wouldn’t necessarily know of that are from Asia like Zapu AI and Baichuan and RWKV. So lots of different partners there. It felt like they were really advancing this LLM capability on-device, and they’re leaning towards these smaller models.
I mentioned 13 billion parameters and the 10 billion parameters. If you look at big LLaMA, parameters are 70 billion. OpenAI is three times that. They’re very, very large. So you had the 6 billion and the 30 billion parameter LLMs, and there was a partner that came up on stage, an OEM partner called Honor, and they said that they had a 6 billion parameter model was really the minimum size model they found for on-device AI with no cloud connection. And the CEO was talking about and he said smaller than that is “just a toy.” So he said, but the bigger these models are, the battery is a limitation. So he’s talking about smartphones and he said you can’t run larger models because of power consumption issues. So Honor has built their own LLM and it’s 6 billion parameters. That’s something to note.
The last thing I’ll say as far as notes from the conference was there was a senior vice president, his name Ziad Asghar, and he spoke very briefly about an idea I’d like to explore that I think is interesting for on-device AI and he said multimodal AI. And that is interesting. So that’s the idea that you marry an LLM model with diffusion models that are creating these fusion language outputs with image and video-based outputs. And that made sense to me when you’re thinking about on-device AI, so smartphones and onboard cameras and what you can do with that. So set that aside. So here’s the quick why I think that this on-device thing is interesting. There’s a few things. One is this opportunity. So let’s go back to that 3 billion number of Snapdragons. The installed base with Snapdragon today is 3 billion, and that’s just a compelling market opportunity. Enterprises are always interested in the size of the market opportunity and speed in which they might be able to generate something within that market opportunity. And when you think about the value proposition of mobile apps, it became so compelling the business opportunity. There was just billions of business. So it was really a great business opportunity. And so when you have enterprises that are interested in AI use cases, they get even more interested when that market opportunity is that large. So if you say you’re an enterprise, you’re thinking about smartphones as the way that you’re going to deliver some product using AI, that becomes really compelling. And I think that it’s just hard for us to ignore that the most popular computer platforms in the world are our smartphones.
And so it’s going to be hard for us to move forward in AI even though it’s really this power crunching thing that we understand it to be today without getting to smartphones and PCs. So think of market opportunity. That’s a really interesting thing. I think that it became evident to me that here we are and who’s going to help us get there. And we saw some of that evidence in saying, well, you have to have really efficient power. You have to have good compute on-device. So here’s Qualcomm setting themselves up. I think they have some competitive advantages going forward into this market because not many players make GPUs in general and even less make them for mobile devices, PCs, and smartphones. So there were four that I could think of is Qualcomm, Intel, AMD, and Apple. And of those that make GPUs for devices, who’s been thinking about it the longest? Well, I think it’s Qualcomm and maybe Apple. So you have this experience they have with putting GPUs on these devices, which is interesting. Lots of other places such as we talk about data centers and all this GPUs were used for different things, but mobile chip makers like Qualcomm have been dealing with GPUs for a while. So that’s really interesting. And it is also interesting, I think there’s a competitive advantage for Qualcomm because they said that they’ve been thinking about and working on AI, which I believe makes sense because of their experience with smartphones for 10 years. That gives them a leg up in thinking about on-device AI, and all the time they’ve been thinking about on-device AI. So the difference has been that we last year introduced this idea of generative AI. So that’s really changed the game. Third competitive advantage for Qualcomm as it feels like they’re really further along in terms of the partnerships and the ecosystems to support on-device AI than some of the others. And that includes their whole stack. They might have the bigger developer ecosystem than some of the others out there that are in the mobile space. And it was evident to me at the summit they had thought pretty deeply about logical use cases for on-device AI. I get a little worried about those because I’m not real sure what they are, but I thought I saw some interesting things. First, they talked about the personal assistant, which I think is a challenge for on-device if you’re not always connected, but we’ll get at that at some other point. It’s an idea. The one that made really a lot of sense to me were these ideas that you have these camera-based use cases where you’re thinking more about collaboration and productivity applications like video conferencing.
So I thought that was interesting and that’s where their competitive advantages are. The challenges for Qualcomm and on-device AI is just really early days for generative AI use cases, and this was mentioned, this limitation of the battery. How much AI compute can smartphones and PCs really do process locally depends on the use case. We don’t know. So I think that most of these on-device use cases are going to require a hybrid approach, which is something that Qualcomm already promotes. So there you go. I think that they set themselves up to be a player in on-device. And it’s early days, like we said, for this generative AI, regardless of it’s cloud-based or it’s on-device. Qualcomm has a huge opportunity in front of them and they appear to be all in on pursuing it. The next two years will reveal a lot about how and where we can use AI in general. My bet is that Qualcomm will have something to say about what on-device looks like and how that becomes reality. All right, so that’s a company we like. That is it for this week and we’re just under the 30 minute mark. I love that. Thanks for joining me here on The AI Moment. Be sure to subscribe and rate and review the podcast on your preferred platform. We’ll see you next time. I’m Mark Beccue. Thanks.
Other insights from The Futurum Group:
Two Trends in AI Regulations and a Look at Microsoft Copilot – The AI Moment, Episode 2
Key Trends in Generative AI – The AI Moment, Episode 1
Author Information
Mark comes to The Futurum Group from Omdia’s Artificial Intelligence practice, where his focus was on natural language and AI use cases.
Previously, Mark worked as a consultant and analyst providing custom and syndicated qualitative market analysis with an emphasis on mobile technology and identifying trends and opportunities for companies like Syniverse and ABI Research. He has been cited by international media outlets including CNBC, The Wall Street Journal, Bloomberg Businessweek, and CNET. Based in Tampa, Florida, Mark is a veteran market research analyst with 25 years of experience interpreting technology business and holds a Bachelor of Science from the University of Florida.
