Whether it's providing real-time support, automating processes, or delivering personalized experiences, Conversational AI has become a key differentiator for organizations seeking to stay ahead in the market. However, harnessing the power of Conversational AI requires a strategic partnership with the right technology provider, making the process of selecting the perfect ally a crucial step towards success.
In this article, we talk with experts and shed light on platform evaluation, an indispensable step on your path to harnessing the full potential of this transformative technology. We’ll be taking you through:
- A Conversational AI Platform 101: where to start and how to know what you’re looking for
- Two experts’s take, i.e. a Product Owner working in a prestigious banking organization, and Amir Kiabi, Digital Transformation & Improvement Manager
Conversational AI Platform Evaluation 101
Identify what you’re looking for in CAI
Before you start venturing into the crowded forest of CAI vendors and frameworks, it is always advisable to take some time to figure out what you want to achieve with CAI, as we discussed in more detail in our guide on How to Find Your Perfect Conversational AI Platform, where we discussed 18 criteria that you might want to consider to guide your choice.
According to the experts, you should be asking yourself:
How do you envision CAI within your company?
Is it a chatbot or a voicebot? Do you see it as both?
Or do you see CAI as a broader capability?
Depending on the purpose you determine, you’ll be able to figure out what kind of supplier you’re looking for. For example:
- If you want to keep it to a simple chatbot, a SaaS provider might suffice.
- If you want to have the bot become an integral part of your digital domains, or digital offering, you might be after a cloud provider, or a full-service CAI platform.
Only upon determining your purpose, will you be able to translate it into needs that your AI specialists and conversation designers will have. In turn, these needs should then be translated into requirements for your IT department, who will take care of your backend integrations and system.
Getting *officially* started
Before you get knees deep in the conversational AI market pond, there are some practicalities to be sorted. What this usually means is that someone needs to create a Demand Specification or Business Case to determine the need, cost, benefits etc.
If you’ve been keeping some interesting technologies and companies that offer CAI solutions on your radar, this is also a good time to start jotting down names.
Selecting the pool of CAI platforms to consider
The goal in the process of selecting which candidates you’ll consider as your potential technology partner is to be as objective as possible. Of course, your hard work of keeping up with the latest trends and emerging vendors shouldn’t be thrown down the drain. The names you’ve noticed can be included in this initial set of considerations.
To stay clear of personal preferences, it helps to look at rankings, like Gartner’s, Forrester’s and G2’s.
From these rankings, you can get a first list of 10-15 potential candidates, which you want to narrow down to 3-5 that you are most interested in and want to get in contact for a demo. This process requires research, meetings with vendors, reflections on technical aspects, like whether the platform can be integrated with your internal systems.
Out of the box or DIY?
The ideal situation would be to find the perfect vendor that suits all your needs and technological requirements and offers all the features you could dream of out of the box.
In reality, this isn’t often the case, especially as your conversational capabilities structure themselves further and mature. The solution you often see in organizations is the arrangement of an ecosystem of tooling. Every tool should be either able to communicate with each other, or produce inputs that are very easy to convert into one unifying platform.
It’s not necessarily that having one platform that checks all your needs with its out-of-the-box features is better, or worse, than relying and creating your custom “DIY” ecosystem of tools. Both are perfectly feasible and workable, and it ultimately comes down to how specific your requirements are and what the CAI vendor market currently offers.
Experts take: CAI Platform Evaluation
Naturally, all experiences of platform evaluation will be different and unique to the specific situation. While some considerations appear to be shared more broadly across organizations, the two perspectives of experts who’ve been through this process highlight also different ways of approaching this pivotal process.
To represent this, we’ve talked to two professionists, a Product Owner working in a prestigious banking organization, and Amir Kiabi, Digital Transformation & Improvement Manager, who have shared their take on CAI platform evaluation and more.
The Perspective of a Product Owner who works in the Banking sector
Who is involved in the process of CAI platform evaluation?
“A wide range of people are involved in this process. Broadly speaking, we can think of two groups:
- People (e.g. Back-end Developers) who’ll be involved in setting up the platform for your organization and making sure it’s hosted securely. Their focus will be on vouching for technical hygiene factors.
- People (e.g., Conversation Designers, Data Scientists/Analysts) who will be working with that specific platform or tool.
The first group needs to align with the second to understand what kind of flexibility they need as internal users of the platform. Doing this in advance allows the technicians to cater designers and scientists properly, ensuring they have enough flexibility to be able to meet business goals.”
Who ultimately makes the call on what platform is chosen?
“Conflict can easily arise between conversion designers, data scientists and backend developers about what platform should be chosen, so ultimately you need to have someone who makes the call. It starts with setting the requirements right from the beginning and establishing a group alignment on how different aspects will be weighed in the evaluation, based on the goals you collectively agree on, as a team.
Down the line, though, you’ll need to have someone, like an Engineering Lead, a Senior Product Owner, a Project Manager, who’s deeply involved in this process and aware of all the conversations around it. They are the ones who will ultimately say in the name of the group: “We choose to procure X.”
What are key criteria to look at for CAI platform evaluation?
“One thing that I want to stress and is often overlooked is the usability of the platform, particularly for conversation designers. If the end users aren’t able to comfortably work with your platform, that is immediately reflected in the quality of your product. That’s because they might end up giving up on their vision, or their vision simply might not end up being translated well into the model.
Compatibility with the preferred cloud platforms, or dedicated data center, is also definitely important.
Generally, though, I try to stay away from technical elimination requirements, and knock-out criteria. When you’re looking for sourcing like this, you’re usually not really just looking for the right platform but also for the right partner. So, it’s really more about asking yourself:
- Does the company you're going to work with have a clear vision on where it's heading in the coming 2-3 years? Is that also aligned with the direction you want to take it?
- Who are the representatives of that company that are going to be involved in your project? The last thing you want to have is a great relationship with a sales team, and once the contract is signed, you get a new team in front of you and you don't know who they are. ”
Should you account for the risk of vendor lock-in, when you first choose what platform to rely on?
“Vendor lock-in is one of the top considerations made.”
Do you take the testing capabilities that a platform offers out of the box into consideration?
“Dialogue testing is separated from the platform for us. In some areas of CAI, you have some really niche players that excel in one thing and are easy to integrate with the generic platform you use. In testing, there can be some specific testing purposes, e.g. figuring out if your dialogues are broken, or generic flow testing before you push something to production. These are specific needs that can be perfectly met only from a niche platform.”
Should analytics and reporting capabilities be considered in the evaluation of generic CAI platforms?
“I don’t think that being able to build dashboards within your platform isn’t important, but I wouldn’t consider it as a fundamental criteria for electing a partner. Analytics is the perfect example of an area where relying on your existing technology stack is often preferable.
In terms of the metrics that are measured in a platform out-of-the-box, I’d recommend paying attention to what the definition that specific vendor uses for that KPI. Something I noticed in the market is that everyone assumes they understand what common metrics, like containment, dropouts, answer given, mean. Often, though, different companies will have different definitions of them and not being aligned with your platform on this can skew your analyses.”
The best KPIs for CAI applications
“It really depends on the purpose of your application. If you build a conversational IVR, containment doesn't matter. If you build a voicebot to help out your contact center agents, then containment is a very important metric. So, it’s really about understanding what’s the purpose of what you’re building and defining tailored KPIs that can measure what you care about.
In general, one thing that’s really important is to define your set of KPIs on the endpoints of your application, for you to understand where the customer journey ended in your application.”
What are your thoughts on NPS, the most dividing metric in the CAI space?
“On the one hand, NPS is really often the only customer feedback metric that sticks organization-wide, so using it enables you to cater to your stakeholders. On the other hand, it’s not really suitable from a practical perspective to assess how successful your bot is.
So, I’d consider it as more of a reporting metric, than a performance one. If you use it, make sure you also bring this up, indicating what would be better alternative metrics to look at to assess performance (e.g. containment, hand-over rate, dropout).”
The Perspective of Amir Kiabi, Digital Transformation & Improvement Manager
Who is involved in the process of CAI platform evaluation?
“This is different from company to company, but usually there is an internal process that usually involves: Digital Transformation Manager, Head of Customer Service, Enterprise Architecture Team, Project Manager, IT Director, Business Analyst, and maybe other roles.”
Who ultimately makes the call on what platform is chosen?
“In the final negotiations it was myself, IT Director, and Head of Business Development (Enterprise Architecture Team)”
What are key criteria to look at for CAI platform evaluation?
“When we get to evaluate different technology vendors, we have a Business Case set up. Here we document all the facts, like price, pros/cons, benefits, integrations, as well as the technical aspects, protocols, interfaces, and additional offers (e.g., do we get a full team with Customer Success Manager, Project Manager, Key Account Manager?).
The key criteria we look at are:
- Technical aspects
- Connection with the company (e.g., where are they based)
- Pricing based on offer
- Integrations.”
Should you account for the risk of vendor lock-in, when you first choose what platform to rely on?
“In this day and age, integrations are crucial for any platform. This was a key reason for choosing Boost.ai for example. They had multiple integrations "out of the box" which were all suited for our demands. Most of the top vendors offer similar integrations, so vendor lock-in is not as big an issue.”
Did you take the testing capabilities that a platform offers out of the box into consideration?
“We didn’t want to rely on external testing tools for testing. We simply connected the Boost.ai platform to an internal page and involved internal resources, all the way from customer service agents, field reps, to IT resources, marketing teams, and high level managers. As this was a live chat, we connected it to a test page and sent out the link, asking everyone to test real life scenarios.
In the Boost.ai admin panel we could then see the % on the good match rate, or success rate, and verified the results with the Boost.ai CSM, PM and their team."
Should analytics and reporting capabilities be considered in the evaluation of generic CAI platforms?
“We had our own metrics as we switched from another platform over to Boost.ai, but, of course, we relied on the data that was presented in the Boost.ai admin panel. Additionally, for the first year, we analyzed and graded every single chat manually.
The best KPIs for CAI applications
“The most important metrics would be:
- Good match rate (or Success rate)
- Automated chats
- CSAT (Customer satisfaction).
We also looked at whether the customer accepted the answer, especially when the bot provided an answer that is correct, but that they might not want to hear. Constantly monitoring the tone of voice of the answers is also crucial, but hard to use as a metric.”
What are your thoughts on NPS, the most dividing metric in the CAI space?
“We did use NPS as a metric, but not specifically for the CAI. However, I believe that when implementing a new solution like a CAI, NPS is heavily recommended, not as a way to measure the CAI itself, but rather to make users aware about the existence of this new channel of communication. For example, you could add a question in the survey saying: “We have recently upgraded our digital communication channels. Have you tried them? If yes, please rate your experience."
This way, customers who haven’t used the chatbot before will become aware of its existence and consider trying it out, the next time they want to connect."
It’s truly interesting to see the perspectives of two people who've been through the process of evaluating and picking the right conversational AI platform for their organization.
We hope it helped to give a concrete idea of what is involved and what are steps that different experts have in common, as opposed to the ones where they might take different paths.
At Vocalime, we understand that the process of evaluating different technology vendors can be overwhelming and complex. That's why we also offer strategic guidance and support to help businesses choose the best technology for their unique requirements.
Looking to figure out how conversational AI can help your business achieve your desired outcomes and drive growth? Feel free to reach out via voice message or good old email.
And if you’re still hungry for knowledge, follow us on LinkedIn for weekly updates on the world of conversational AI, or check out our article about how to find your perfect conversational AI platform, where we share our top 18 criteria to guide your choice.
