The News: According to a paper published by researchers at MIT Sloan and Stanford University, inexperienced contact center workers stand to benefit the most from the use of generative AI. MIT Sloan associate professor Danielle Li, MIT Sloan PhD candidate Lindsey Raymond, and Stanford University professor Erik Brynjolfsson, PhD., studied the staggered introduction of a ChatGPT-based conversational assistant using data from 5,179 customer support agents, and found that access to the AI tool increased agent productivity, improved customer sentiment, and helped drive transfers to other departments earlier in the chat interaction.
The AI tool monitored customer chats and provided agents with real-time suggestions for how to respond, though agents were free to ignore its suggestions. Workers using the generative AI tool increased the number of customer chats resolved per hour by 13.8%, the researchers found. In addition, workers using the AI tool spent an average of 35 minutes on each chat, compared with 40 minutes for their colleagues who did not use the model. These figures also account for the fact that workers often manage more than one chat at once.
The productivity gains were highest among workers with the least experience, who resolved 35% more chats per hour when they used the AI tool, while productivity was essentially flat for workers with the most skills and experience who used it.
Using the generative AI tool also led to improved customer sentiment. Requests to speak to a manager declined by 25%, and AI assistance markedly improves how customers treat agents, as measured by the sentiments of their chat messages. Further, transfers to other departments tended to happen earlier in the chat conversation, which the researchers suggest was due to the AI tool being able to help agents more quickly match a customer’s problem to the right business unit for a solution.
The AI tool offered recommendations only if it was “sufficiently confident” in its answers, which reduced the number of incorrect responses, according to the researchers, who added that workers were not required to use the recommendations. Agents followed the recommendations 38% of the time, which the researchers said was consistent with the industry average for generative AI tools.
You can read the study at this link.
Study: Generative AI Is Most Helpful to Less-Skilled Contact Center Workers
Analyst Take: To study the impact of large language models (LLMs), researchers from MIT Sloan and Stanford University reviewed the use of a ChatGPT-based conversational assistant that was used by 5,179 customer support agents serving a Fortune 500 company that sells software to small businesses in the US. The researchers found that the AI tool increased worker productivity, resulting in a 13.8% increase in the number of chats that an agent can successfully resolve per hour, and noted that three components of productivity were impacted: a decline in the time it takes to an agent to handle an individual chat, an increase in the number of chats that an agent can handle per hour (agents were able to handle multiple calls at once), and a small increase in the share of chats that were successfully resolved.
Greater Impact With Less Skilled and Less Experienced Workers
The researchers also found that the AI tool disproportionately increased the performance of less-skilled and less-experienced workers across all productivity measures, and it helped newer agents move more quickly along the experience curve. Indeed, researchers found that AI-assisted agents with 2 months of tenure perform just as well as unassisted agents with more than 6 months of tenure.
The reason for this finding is likely due to the ability of a generative AI tool to quickly distill the months or years of knowledge and experience that is accumulated by experience agents in a digestible format that new agents can quickly use. This is especially valuable to contact centers, which often have little time or resources to adequately train new agents, nor the structure to assist experienced agents with mentoring or guiding their newer colleagues. For this reason alone, contact centers should consider using generative AI tools that summarize content, suggest potential responses, and otherwise tap into the knowledge accumulated by its agent base and institutional knowledge stores.
Helping Customers Get to the Right Resource More Quickly
The ability to quickly understand customer inquiry intent is a massive determinant of customer experience, largely due to the amount of frustration incurred by customers who know they are not able to be helped by a front-line agent, and simply wish to be directed to someone who can handle their issue. The use of the generative AI tool was viewed as a catalyst for helping an agent quickly ascertain if the customer needed to be transferred to a different department or agent, based on the information provided by the customer, and the responses suggested by the AI tool, reflecting significant value as a tool to help identify and distill customer intent.
Reducing the Need for Managers to Intervene
In a customer support situation, manager intervention usually means a customer is unable to have their issue resolved, and therefore will be unsatisfied and perhaps angry. By utilizing the AI tool, the researchers found that the requests to speak with a manager declined by 25%. This allows the managers to focus on only the most serious or pressing issues. With fewer requests to handle, managers can more clearly focus on each interaction, improving resolutions and customer satisfaction. Further, as the number and frequency of manager interventions declines, it should afford the manager time to focus on staff development, training, or other administrative tasks that are often relegated to the back burner.
Will Generative AI Tools Have a Greater Impact on Experience Workers as More Complex Data is Introduced?
An open question is whether the deployment of generative AI tools that are designed to handle more complex data or inquiries will result in productivity gains for experienced and knowledgeable workers. Even 6 months of experience in handling the same type of customer service inquiries repeatedly will allow most workers’ recall ability to match or exceed an AI tool’s abilities with basic inquiries or requests.
However, as organizations become more familiar and comfortable with generative AI tools, and allow the models to be trained on more complex or detailed data, it is likely that even experienced agents may find value in using these tools.
Specifically, I can see experienced agents working in the financial services, healthcare, and insurance industries, where precise, accurate, and complete responses to questions are critical to the customer’s experience and compliance with regulations, finding significant value in a generative AI tool. The catch, of course, is ensuring that the model is trained or grounded in the right data, and that appropriate guardrails are put in place to ensure that the model does not hallucinate.
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Other insights from The Futurum Group:
AMD Datacenter & AI Conference Recap: All Eyes on AI and Cost Optimization
HPE Enters the Era of AI With New Cloud Service
Despite a Rise in AI in CX, People Still Prefer to Interact with Humans
Author Information
Keith Kirkpatrick is VP & Research Director, Enterprise Software & Digital Workflows for The Futurum Group. Keith has over 25 years of experience in research, marketing, and consulting-based fields.
He has authored in-depth reports and market forecast studies covering artificial intelligence, biometrics, data analytics, robotics, high performance computing, and quantum computing, with a specific focus on the use of these technologies within large enterprise organizations and SMBs. He has also established strong working relationships with the international technology vendor community and is a frequent speaker at industry conferences and events.
In his career as a financial and technology journalist he has written for national and trade publications, including BusinessWeek, CNBC.com, Investment Dealers’ Digest, The Red Herring, The Communications of the ACM, and Mobile Computing & Communications, among others.
He is a member of the Association of Independent Information Professionals (AIIP).
Keith holds dual Bachelor of Arts degrees in Magazine Journalism and Sociology from Syracuse University.
