Evaluating Chatbots for Customer Support Applications

Chatbots Are Evolving From Rules-Based Systems to Understanding Language and Intent, Increasing Functionality and Requiring More Assurance of Proper Functioning

chatbot customer support

A key technology used to aid in call deflection within a contact center is the chatbot, a text or voice-based interface that is deployed on the website or within an application to simulate conversation with users and seamlessly support users. Many of these are self-learning bots, and use natural language processing (NLP) and machine learning (ML) technologies, which allow the bot to converse with customers in a conversational tone, providing answers and solutions to a range of customer inquiries, problems, or service issues.

The bots of today can automate tasks, understand words and phrases, frame appropriate responses, and learn from the received inputs, allowing organizations to replace or augment human workers, particularly for common or relatively simple tasks. Some of the key players in the space include Nuance, Amelia, Verint, Kore, Inbenta, Artificial Solutions, and others, along with any number of CX platform providers, which often incorporate AI-based chatbot functionality. For companies seeking to build their own chatbots, companies such as Google Dialogflow, Amazon Lex, IBM Watson Assistant, Facebook’s Wit.ai, and Microsoft Azure Bot Services provide the tools and platforms to create customized virtual assistants.

Because AI-based bots can understand language and intent, they have significantly more functionality and polish than traditional, rule-based bots. These bots can perform various tasks, such as conducting sentiment analysis, predicting consumer likes and dislikes, assisting the customer with the right product or service, and, if a stumbling block occurs, can quickly route the inquiry to a live human for resolution.

Not all chatbots are created equal, however, so several assessments should be made to ensure that the bot technology that is being purchased, modified, or built from scratch internally can meet the following attributes:

  • Responsiveness: An artificial intelligence (AI) conversational bot should be able to reply fast as soon as it receives inputs from the user, which portends the use of streamlined backend integrations with various data sources and applications, to reduce or eliminate the time it takes for a bot to respond to a customer’s input.
  • Response accuracy: Responsiveness alone is worthless if the accuracy of the responses or answers that bots provide is low, unclear, or incomplete to the end user. Further, the bot must be tested to ensure that not only are the appropriate responses to queries returned, but that the bot is properly tuned to ignore inappropriate, insensitive, or objectionable inputs that may elicit unwanted responses.
  • Clarity and error handling: A bot’s ability to deal with the errors and its ability to recover from those errors must be tested, so that if a bot fails to understand user inputs, it can ask the customer alternative questions for clarification, or should immediately connect the user with a live agent.
  • Tone and personality: A bot’s voice (whether deployed via voice or text) should be relatable and fit within the character and tone of the user base, so that users feel comfortable conversing in a normal manner. A bot deployed within a financial services use case is likely going to differ in tone and feel from a bot deployed via a streaming music service.
  • Ease of navigation: The navigation flow of a bot should be tested to ensure the customer does not feel lost while speaking with the chatbot, and can easily access the information without navigating through decision tree-like menus.
  • Recall and intelligence: Abot’s intelligence must be tested to ensure it can recall the information presented to it by the customer, and then provide the correct response, using a combination of knowledge bases, supervised ML, and a single source-of-truth for important policies or procedures.
  • Understanding: A bot should be able to understand all requests, small talk, vernacular, idioms, and emojis sent by the user to frame appropriate responses.
  • Omnichannel and device compatibility: A bot must be able to perform seamlessly across all devices, platforms, and OS versions, and communication channels, with a similar experience and feel. Customers should be able to seamlessly switch to another bot or a live human without needing to re-identify themselves, or repeat information.
  • Multithread understanding and execution: Customers often have more than one task to accomplish, and bots should be able to identify multi-process and non-related separate queries, and seamlessly be able to handle them in a logical manner. This is essential to creating bots that can more efficiently handle more complex inquiries and further improve live call deflection rates.
  • Security: Data security is a major concern for all enterprises, and bots should be able to be integrated within the organization’s security framework to ensure the data being provided to the bot is properly handled and secured, particularly when it is regulated information (such as healthcare or financial data). Organizations should deploy frequent security testing to find and eliminate security loopholes or vulnerability issues.

Author Information

Keith Kirkpatrick is VP & Research Director, Enterprise Software & Digital Workflows for The Futurum Group. Keith has over 25 years of experience in research, marketing, and consulting-based fields.

He has authored in-depth reports and market forecast studies covering artificial intelligence, biometrics, data analytics, robotics, high performance computing, and quantum computing, with a specific focus on the use of these technologies within large enterprise organizations and SMBs. He has also established strong working relationships with the international technology vendor community and is a frequent speaker at industry conferences and events.

In his career as a financial and technology journalist he has written for national and trade publications, including BusinessWeek, CNBC.com, Investment Dealers’ Digest, The Red Herring, The Communications of the ACM, and Mobile Computing & Communications, among others.

He is a member of the Association of Independent Information Professionals (AIIP).

Keith holds dual Bachelor of Arts degrees in Magazine Journalism and Sociology from Syracuse University.

Latest Insights:
How Genesys and AWS Are Redefining AI-Driven Customer Engagement
July 24, 2026
Article
Article

How Genesys and AWS Are Redefining AI-Driven Customer Engagement

Keith Kirkpatrick, Vice President & Research Director, Enterprise Software & Di at Futurum, Genesys Cloud's expanded AWS partnership leverages agentic AI to transform enterprise customer engagement and enable autonomous interactions at scale....
So This Is How AIs Attack- Observations From the OpenAI & Hugging Face Incident
July 24, 2026
Article
Article

So This Is How AIs Attack: Observations From the OpenAI & Hugging Face Incident

Fernando Montenegro and Mitch Ashley, VPs at Futurum, read the OpenAI and Hugging Face agentic incident as a live test of enterprise readiness to detect and contain AI agents that go off-goal...
Nokia Q2 FY 2026 Optical Strength Positions Nokia for the AI Buildout
July 24, 2026
Article
Article

Nokia Q2 FY 2026: Optical Strength Positions Nokia for the AI Buildout

Futurum Research analyzes Nokia’s Q2 FY 2026 earnings, focusing on AI and cloud order strength, optical and IP demand, AI-RAN timing, and supply constraints....
Texas Instruments Q2 FY 2026 Earnings Climb on Broad-based Analog Growth
July 24, 2026
Article
Article

Texas Instruments Q2 FY 2026 Earnings Climb on Broad-based Analog Growth

Brendan Burke, Research Director at Futurum, reviews Texas Instruments' Q2 FY 2026 earnings, the broad industrial recovery, doubling data center revenue, and automotive re-acceleration....
Latest Research:
Redefining Creative Workflows in the AI Era
July 21, 2026

Redefining Creative Workflows in the AI Era

In Redefining Creative Workflows in the AI Era, completed in partnership with Adobe, Futurum Research examines how organizations are moving beyond isolated AI productivity gains toward unified creative platforms that...
The Governance Gap Why Scaling AI Requires More Than Monitoring
July 15, 2026

The Governance Gap: Why Scaling AI Requires More Than Monitoring

In our latest market brieft, The Governance Gap: Why Scaling AI Requires More Than Monitoring, completed in partnership with IBM, Futurum Research examines why AI governance must evolve from periodic...
The Rise of the Super Agent: How Agentic AI Is Reshaping the Enterprise
July 13, 2026
Research
Research

The Rise of the Super Agent: How Agentic AI Is Reshaping the Enterprise

In our latest Market Brief, The Rise of the Super Agent: How Agentic AI Is Reshaping the Enterprise, completed in partnership with Lenovo, Futurum Research examines how autonomous AI systems...

Book a Demo

Welcome

The vision behind everything in Futurum’s Custom Research practice is this: research should show you what is happening, what comes next, and what to do about it. It should be personal to each audience, easy for people to grasp, and structured so LLMs can reason over it accurately. And it should be fast and turnkey; you want answers now, not another project to carry for quarters.

Whether you are defining business, channel, or go-to-market strategy; evaluating vendors or justifying ROI; or commissioning research to fill an emerging market need, we have your back, with a program that answers your questions with the objectivity and credibility to drive real decisions.

To do it, we bring unmatched data to bear: Futurum research, surveys, and market projections; validated market feeds; ETR’s 15 years of insight from 10,000 technology decision-makers; G2’s buyer and user data; and what our analysts hear every day. Add leading primary collection, from AI-moderated voice interviews to surveys and analyst-led interviews, all turnkey, and every project comes out credible, nuanced, and actionable.

And we don’t just drop the results in your lap. For internal work, we provide analyst-led sessions, interactive dashboards, and a range of formats. For market-facing work, Futurum delivers turnkey activation and amplification that actually gets seen, by people and by LLMs, through our media and share of voice. This is research that moves decisions and markets.

We will meet you wherever you are, from a fast-turn brief to a multi-year program, and shape the work to your goals, timeline, and budget. The right program for your moment.

If any of this is useful, I would love to talk.

Benjamin Brown, VP Custom Research, Futurum Research

Benjamin Brown

VP, Custom Research · The Futurum Group

Newsletter Sign-up Form

Get important insights straight to your inbox, receive first looks at eBooks, exclusive event invitations, custom content, and more. We promise not to spam you or sell your name to anyone. You can always unsubscribe at any time.

All fields are required






Thank you, we received your request, a member of our team will be in contact with you.