On this episode of The AI Moment, we will take a deeper look at a key trend in generative AI – the continued emergence of on-device AI, with special guest Olivier Blanchard, Senior Analyst and Research Director at The Futurum Group.
The discussion covers:
- Key Trends in Generative AI – The installed base of smartphones and PCs is about 6.42 billion – 4.45 billion smartphones, 1.97 billion laptops, tablets and desktop PCs. This is a massive market opportunity to leverage generative AI applications. But there are challenges, including limitations to local processing, battery power and connectivity. I walk through some of the challenges and opportunities with my colleague, Olivier Blanchard, Senior Analyst and Research Director at The Futurum Group, whose practice area includes devices.
Watch the video below, and be sure to subscribe to our YouTube channel, so you never miss an episode.
Listen to the audio here:
Or grab the audio on your favorite podcast platform below:
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this webcast.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Transcript:
Mark Beccue: Hello, I’m Mark Beccue, Research Director for AI at The Futurum Group. Welcome to The AI Moment, our weekly podcast that explores the latest developments in enterprise AI. I don’t have to tell you the pace of change in AI has been unprecedented. I’ve been covering AI since 2016. Never seen anything like what we’ve experienced since ChatGPT launched about this time last year and kick-started the generative AI era. And that’s why we call this The AI Moment. What we’re trying to do with this podcast is distill the mountain of information that’s just comes at us at this pace that’s incredible, separate the real from the hype and provide you with sure-handed analysis about where the market will go. Typically, we dive deep into the latest trends or technologies and what’s shaping the AI landscape. So today we’ll be covering the key trends in generative AI and we’re going to be talking about on-device AI with a special guest, Olivier Blanchard. So I’m going to set that up before I introduce him and say this.
Been thinking about on-device AI for a few months now, and here’s the interesting foundations to that. The install base of smartphones and PCs is estimated to be about 6.4 billion devices. That’s 4.4 billion smartphones and about 1.9 billion laptops, tablets or desktop PCs. And that’s a huge install base, massive opportunity. And a lot of generative AI folks or just enterprise in general are thinking about that install base and what they might be able to do and see that as market opportunity. So huge market opportunity. But there are some challenges to on-device AI, a few just to mention right off the top. Limited compute power. The chips that are used for these devices are only so powerful when you compare them to what typical AI workloads are like right now these days are mostly running in the cloud on these huge compute supercomputers and things like that. So you have to think about that. Power usage is an issue. Battery challenges, especially for smartphones to run AI workloads. And then this idea that you may not always have connectivity. So there is a element to AI on on-device that would need to be at least partially local compute. So when you look at all that, those are some of the challenges. But there are a lot of interested parties in promoting on-device AI, including the chip makers and the device OEMs. So let’s explore this. So I said I have a special guest. I’m joined today by Olivier Blanchard. He’s Senior Analyst and Research Director here at The Futurum Group. His practice area includes the coverage of some of these players and he’s here to share a bit today. So hello Olivier, welcome to the show. And why don’t you start by telling us a little about your coverage area.
Olivier Blanchard: Hi, nice to be here. First timer. So I’m really excited about the show. I love AI. So I’m Olivier Blanchard, I’m Research Director at The Futurum Group. And one of my coverage areas that is kind of relevant to this one is I cover devices. So whether it’s watches, PCs phones, any IoT, smart glasses, all of that stuff generally falls under my purview. So there’s also a piece of this that’s automotive, which to me is also a device now and also plays into the AI space. And I also focus on a lot of policy and regulations. I leave most of the AI stuff to you obviously, because that’s your purview. But there’s a little bit of overlap with, I cover the European Commission and regulators for a lot of different markets. So AI obviously is a hot button topic these days that at least I stay aware of it and let you handle that.
Mark Beccue: Gotcha. So let’s start with this market opportunity and it kind of laid out, gave you just some statistics there. I want to just get your opinion on what do you see as compelling about that, those numbers?
Olivier Blanchard: So let me go back in time about a year, because we’ve only been dealing with this for about a year, right? And it’s crazy how far and how fast we’ve gotten already with generative AI. But I remember that a year ago, the conversation that I was having with enterprise players especially and some of the chip makers who shipped to them was, okay, so we’re going to get this huge demand for generative AI and we’re going to need data centers that are adapted to this particular use case. So that means a lot more GPUs. It means stringing GPUs together in ways that makes them more efficient, more power efficient, but also just more efficient with inference and compute. There’s also a huge CPU angle that doesn’t get as much attention as the GPUs for some reason. And all of that at scale was going to be really difficult. So there were supply chain issues, there were just the engineering for these types of situations and setups didn’t really exist yet, or at least not at a level that could sustain the demand. There was a real structural challenge of we can build this out as fast as we need to.
So I think that some of the chipmakers who already were working in the device space saw an opportunity there. And one of them is Qualcomm, for instance. Qualcomm has been kind of like the king of the hill when it comes to mobile chips and also chips for XR. So most of the headsets that you have for VR and augmented reality have Qualcomm chips and Qualcomm IP in them, just like phones, just watches, just like laptops. And so these folks, especially at Qualcomm, have already been working on inserting AI features into their chipsets and into their SoCs, their system on chip, that were being used in mobile and other applications. And one of the particularities of Qualcomm is not only did they do that well, they did that also just better than anybody else in terms of performance, but also with extremely low battery usage. So essentially, their chips not only are really good at AI and performance in general, they also do that at very low power. And so this was an opportunity that I’m not sure Qualcomm had anticipated. I think they just wanted to build AI into their systems just because AI makes your cameras work better. AI creates more cool user experiences, like a digital assistant and translation without necessarily needing a connection. And all of a sudden they’re there and they have the basis, basically the foundations for this piece of it, which is, oh wait a minute, it doesn’t all have to happen in data centers, it doesn’t all have to happen in the cloud. A lot of this stuff can be offloaded on-prem, on device, on the edge essentially, and it can work on your phone, it can be in your laptop, it can be in your other devices. You don’t have to depend on the cloud for this.
Mark Beccue: Right. I was kind of impressed with that, almost like in the way you mentioned that it’s kind of neat that the chip makers or the chip makers for smartphones almost stumbled into this the same way NVIDIA did in that they were the only ones that had these chips that had GPUs in them and CPUs because they were using them for something else. So does that make sense to you that they’re kind of like, well, we already have this purpose-built thing that’s built for efficiency and all this and now we can leverage into AI. Is that how you’re kind of seeing them?
Olivier Blanchard: I think so, yeah. I’ve had a lot of conversations with them in the past and AI was always kind a thing. It was always part of what they were doing. But the explosion of generative AI, the way it just hit the market and just, it was like, this might be insensitive, but it was kind of like a tsunami, in terms of how quickly it took the market by surprise that wait a minute, this is real, this actually does this. And we still have a long way to go in terms of AI is not making a mistake, the generative AI getting better, but it’s already in the past year gotten super good. And I think what NVIDIA and Qualcomm and chip makers who were already good at making the types of chips, whether they’re GPUs or they have neural processors built in, I think we’re really focusing on kind of surface AI, where again, it’s voice recognition, quick translation from voice to text and text to voice. So translation works in the same way generally. And also I think for gaming, and this is where there’s an intersection between NVIDIA, which makes just amazing GPUs for gaming obviously, and Qualcomm that’s been focusing on bringing gaming, like high-end gaming, into the mobile space with their chips. And I think just the architecture of the GPUs and the types of use cases that they were already building around these chips or for these chips even lent itself super well to the kind of workloads that generative AI demands. So they were just having the right product at the right time.
Mark Beccue: So you’re skipping into another question, which let’s just go with that. I was thinking that clearly the use cases are a little different, you’re already alluding to that. And I heard some talk about the idea that there’s a camera in these devices might be something that, in other words, there’s been a slew of different kinds of generative AI use cases out there. Do they really apply to these devices? Sure, a little bit. But really, thinking about what’s on those devices right now, do you think there’s going to be this… first, do you think there are different use cases really for on-device AI? And do you agree with that? And if you do, what do you think other than the ones you mentioned some about gaming and other that, what do you see as being the compelling on device for AI?
Olivier Blanchard: So I try to generally split AI use cases in terms of sort of front of the house AI and back of the house AI. So back of the house AI is the stuff that’s definitely just AI powered, but that you don’t necessarily-
Mark Beccue: Under the hood.
Olivier Blanchard: Right, under the hood. So it’s essentially it’s your camera works better because AI is doing stuff in the background that you don’t see. You don’t get to experience it, you just see the end product. It just sort of magically works. I’m not that good of a photographer. My pictures are getting really good because AI in the background is making them better. That sort of thing. There’s AI built into devices to manage 5G connections, for instance, which are like 5G RFFE and antennas are extremely complex. And so you need AI in the background to manage this and also manage power efficiency of the devices. So you don’t use that power where you don’t need to. So there’s a lot of built-in AI that you’re never going to get to experience that’s already in there. And then there’s the sort of the front of the house, front of the story AI, which is the stuff that we actually experienced that’s UX. And so I think that’s where generative AI has sort of captured the imagination of users and the interest of investors and startups because they can perform tasks for us that we ask them to perform. And so just being able to talk to a device, I am not an iPhone user. I have an Android, and the voice recognition on my Android is stellar. I use it all the time. I ask questions and answers. It tells me jokes, it tells me what the weather is, it automatically knows what news I want to hear, and it reads me the news. It just does really cool stuff like this.
One really interesting application of this sort of voice interface replacing the screen and the keyboard is smart glasses. So Ray-Ban came out with some smart glasses not too long ago in partnership with Meta, and I think they do use Qualcomm or Snapdragon chips in there. And essentially they’re not an augmented reality sort of heads up display that folds in with regular space, but there is some voice activation in there already built in. So you’re telling your glasses to take a picture or to start filming or do something else. And I think that’s sort of the first layer in my view of where generative AI and just AI on devices specifically becomes super important because you can multitask a lot better. You could be typing something, you could be working on something and talking to your computer, talking to your phone or whatever device you have and asking it to perform tasks for you, schedule an appointment, read an email, whatever.
And so I think the value of that already in terms of productivity, in terms of user experience, in terms of building value for devices, especially in a market where PCs have been down for the past year, mobile phones are picking up, actually they’re doing better already. But a lot of consumers are stretched thin with their budgets. Technology has been improving incrementally, year after year after year. There’s not a huge difference between iPhone 13, iPhone 14, iPhone 15. It’s sort of like 25% better, yada, yada, yada. When you inject generative AI and these AI type experiences and interfaces into devices, suddenly, you’re bringing in an entire new set of value for consumers and for IT decision makers, so in the enterprise, if I’m looking at last year’s PCs to upgrade my fleet of PCs because they’re getting old or I’m looking at next year’s PCs that are going to be AI PCs that can do all of these other things, I’m thinking, okay, why would I invest in last year’s technology when I can wait six months and invest in the next decade’s technology?
Mark Beccue: Let me throw one out to you. Talking to a couple of OEMs. One was a PC OEM, a very large one here in the United States, and then a very large chipmaker here in the United States for smartphones. And they mentioned it’s almost scary that this boring in a way, but it was that the camera in both of those devices. With the PC, they’re talking about moving into this better camera MIDI, I think it’s called. Yes. Does that sound right? Its got a new camera. And then the idea around generative AI was to improve collaboration when you need it. So the collaboration tools get better because of the video capabilities get enhanced. And I don’t know how, but they were saying, well, we’re going to use that to… if you think about what Zoom’s doing right now online Cisco, some of the other collaboration players are doing. So it gets a boost. And for instance, let’s say you’re on the road and you need to do a Zoom. We’ve always done that, you’ve got your phone. Sometimes it works well, sometimes it doesn’t, but generative AI might give those already their applications a boost. Have you heard these? So I’m thinking that’s that embedded part you were talking about a little bit, but one of the really compelling use cases might be collaboration.
Olivier Blanchard: I think so. One of the cool things that it hasn’t really taken off yet, I’m really surprised, but one of the things I saw two years ago already was this fun little app that just redirects your eye direction, right? Right now I’m looking at you and me on my screen, but my camera is up here. What it does is it automatically, even though I’m looking down or I might be looking down on my notes or whatever, the eyes are replicated so that I’m always looking straight at the camera, regardless of where my eyes-
Mark Beccue: You need that right now.
Olivier Blanchard: And I think that should be built into every laptop personally. That’s another thing. But I think we’ve almost reaching the point, I think we’ve reached that point where say I have bad connectivity and I can do audio, but I don’t necessarily want to do a video for whatever reason. So maybe my connection’s not super great, maybe I’m in my pajamas, maybe I’m just in a weird environment. There’s no reason why an AI generated version of me, photorealistic version of me that can move my little mouth and move my eyes and do micro gestures to look, get past the uncanny valley phase of this. There’s no reason why there couldn’t be an almost perfect AI generated version of me or you on this call right now doing this. I think that’s fascinating. So it’s a little bit creepy. On the one hand, there’s a huge portion of all of this generative AI that worries me a little bit, and that’s a little bit creepy and dangerous. I think it’s kind of like deep fakes for justice, deep fakes for good. And I think for collaboration, even though there’s already glass between us and now there’s going to be AI generated version like avatars between us, if the option is no video versus video of a version of me that looks real enough, I think that can help us connect better and collaborate better, even if I’m in the car or I’m not presentable, whatever. There’s light and dark. There’s a good use case for every terrible use case.
Mark Beccue: Oh, that’s for sure.
Olivier Blanchard: And so I don’t want to think of some of those kind of potentially dangerous AI use cases as always bad and we should ban them. I’d just rather steer the conversation towards, Hey, how could we actually minimize the negative impact through legislation, whatever, but how can we actually use these applications for good, whether it’s collaboration or healthcare or mental health. There’s a ton of really interesting applications for this kind of stuff.
Mark Beccue: There really is. And I guess time will tell, we’ll see what happens when there’s a lot of discussion around development platforms and the experimentation there. I’ve always said that the model that for purposes of launching a new idea, the app store that came out with Apple was here’s our technology, let people be creative about what they come up with. And in a way, I think that’s going to be the case, particularly with smartphones. You would think that’s the way we go with this, where you get smarter and smarter. You’re putting some sort of tools in the hands of developers and let them be creative about how they would use certain capabilities and what they come up with. So maybe that’s how it goes, at least for smartphones. I’ve also seen that with PCs a little bit. Have you heard that too?
Olivier Blanchard: Yeah. So the AI PC I think is going to be a big deal. There’ve been a few false starts with Arm-based PCs already. I think we’re kind of there now. I think we’re getting to the point where they’re coming out this year, they look pretty solid. Again, time will tell, they’ll keep getting better, but I think we’ve overcome most of the really bad hurdles. So here’s what’s going to be really cool about these PCs. So they have built-in AI, so you don’t need to rely on connectivity. So you could be on a plane, you could be in a place where your internet goes out, whatever. Wherever you are in the middle of the forest with your laptop, you can use generative AI just like you would if you had a connection. So that’s already big. And when I’m talking about this, I’m talking about actually running LLMs on your PC. And so there are two value adds to this. One is connectivity is not a factor. So if you lose connectivity, you don’t lose productivity. You can still keep working through whatever outages or situation that you’re in. But the other thing is also privacy. So when you’re entering prompts and you’re entering data into a generative AI system, it goes out to a data center somewhere, several data centers, it gets processed, it comes back. That data’s vulnerable. Even if you have top-line security, if everything’s solid, there’s still a vulnerability there. The ability to run LLMs and to build your own generative AI models on your PC allows you to insulate yourself and insulate your data and keep it private. And I think that’s going to be a huge value as well for individuals with small businesses, but also for IT decision-makers who are looking at data breaches and how to make their systems more secure, especially with workers who might be off-site most of the time, who aren’t behind a firewall.
Mark Beccue: I’ll give you this. Apparently, so that’ll work in certain use cases and it won’t in others. So I would say it’s going to be difficult for them to do that in the, let’s call it the personal assistance space, depending on how updated that information needs to be. But there are other use cases where it will make sense. Let me do this. We’ve got about five more minutes. I want to ask you your perspective on something else. If you’re thinking about the vendors that are in the space that could benefit from this and all the ones that are playing, who do you think will lead, be a leader in on device AI, if you were just picking one or two or three?
Olivier Blanchard: Are you talking about chip makers? Are you talking about PCOEM?
Mark Beccue: Could be anything.
Olivier Blanchard: I think every PCOEM is on this. They’re all working on it, or at least all the major ones. Obviously, HP, Dell, Lenovo. That’s kind of a given. On the phone side, basically anybody who already… all the smartphones above $800 and that price point have AI on them. I think obviously, the premium sort of extremely high-end flagship smartphones will all have pretty powerful generative AI built in next year already, except maybe Apple. I don’t know, they use their own thing. Their chips are really good, but I don’t know that they focused on AI or at least generated AI as much as the Android and PC market have. So we’re in a weird situation where Apple isn’t necessarily behind, they’ve just approached the problem differently. And judging by the investments that they’ve made in the last two years, I think that they’re working really hard to catch up.
So even if they’re a little bit behind in terms of performance and capabilities on iPhones next year, I think they won’t be too far behind. So I think obviously, Samsung is going to be a huge player in the device space as well for a lot of AI or on-device AI. And then obviously Meta. I think Meta is one of the ones that we forget about. There’s so much AI built into, whether they’re smart glasses, AR or VR headsets. And I think that the voice activation, the ability to interface with your goggles or your glasses by voice using spatial interfaces less is going to be a huge factor. And obviously, being able to bring generative AI into a virtual three-dimension environment is kind of mind-bending and game-changing for designers, for artists, for children, for educators, for engineers, chemical engineers, structural engineers. It’s just the potential is really, really strong. And I know that all the headset makers are working on trying to build more value for the space, for the XR space to generate a little bit more demand than they’ve been seeing. It’s been a little bit anemic for the spatial computing, the face-worn computers. This could be a game-changer for them, but probably two, three years for that. It’ll be phones and PCs.
Mark Beccue: Right. And it’s interesting you mentioned Meta. I have met with a couple of their executives recently that run the Llama side, so the open source folks. And I asked him, I said, “Why are you doing this?” And they said, “Well, remember, we have a lot, these applications that we run, we’re looking to build into our… we’re looking to how we leverage AI into these things. And we learned so much.” He says, “The ROI for us is we’re not making any money by building these models, these Llama models. But what it’s doing is teaching us about how we can do better with our applications once they’re on these devices that you’re talking about.” So they’re actually using it as this massive experiment to make their applications better. Isn’t that crazy?
Olivier Blanchard: It makes sense to me. It’s been funny with Meta especially to see where Mark Zuckerberg was with his vision of the Metaverse two years ago, and how I was just scratching my head at it. A lot of people were on board and saying it was fantastic, and I’m thinking, “Why? This doesn’t make any sense to me, and this isn’t good for Meta.” And since, I don’t know, about a year ago, it flipped. I don’t know if it’s because he got into jiu-jitsu, but he’s in a much better place now as a CEO and I think he gets it. So the Metaverse isn’t dead, but the Metaverse isn’t the point. The point was to bring AI into headsets. The Metaverse comes later when we can build the infrastructure and the use cases for it. And so focusing on the devices first and the experiences and essentially the capabilities of the devices, making them easier to use, making them more valuable, making people actually want to buy them because there’s a point, it’s not something they’re going to spend a lot of money on, buy enjoy for three weeks and then put on a shelf and never touch again. I think that was the point. And AI is sort of the secret sauce for the value of smart glasses and augmented reality. And honestly, being able to have this more just intuitive natural interaction with our devices and our environments is sort of the holy grail of adoption. If something’s difficult to use or I have to learn all these gestures and keep tabs of where my controllers are and keep batteries in them, whatever, that’s a lot of friction. It’s going to make it really hard for me to want to use it all the time. But if it’s something that I can just pick up and talk to or communicate with simple gestures, then it’s something I’m going to want to keep using.
Mark Beccue: And that’s been an argument they’ve been making for a long time. But that’s an argument for another day, right? This is when you talk about all of those fun-
Olivier Blanchard: Yeah, I’m sure I’ll be back.
Mark Beccue: Yeah. All right, well, Olivier, thank you for that today. We’re going to wrap it up here. I’d like to say thank you again, Olivier, for joining us. Thank you all of you for listening and watching, joining us here on The AI Moment. Be sure to subscribe, rate and review the podcast on your preferred platform, and we’ll see you next time. Thanks, this is Mark.
Other Insights from The Futurum Group:
AI Chip Trends, RAG vs. Fine-Tuning, AI2
On-Device AI – The AI Moment
Author Information
Mark comes to The Futurum Group from Omdia’s Artificial Intelligence practice, where his focus was on natural language and AI use cases.
Previously, Mark worked as a consultant and analyst providing custom and syndicated qualitative market analysis with an emphasis on mobile technology and identifying trends and opportunities for companies like Syniverse and ABI Research. He has been cited by international media outlets including CNBC, The Wall Street Journal, Bloomberg Businessweek, and CNET. Based in Tampa, Florida, Mark is a veteran market research analyst with 25 years of experience interpreting technology business and holds a Bachelor of Science from the University of Florida.
Olivier Blanchard has extensive experience managing product innovation, technology adoption, digital integration, and change management for industry leaders in the B2B, B2C, B2G sectors, and the IT channel. His passion is helping decision-makers and their organizations understand the many risks and opportunities of technology-driven disruption, and leverage innovation to build stronger, better, more competitive companies.
