In the last few years, the world of conversational AI has become increasingly interested and reliant on large language models (LLMs). These powerful algorithms can not only make the process of building bots more efficient, but also take over in responding to the user in customer-facing applications.

Efficiency, efficacy, giving more contextual and personalized answers, reduction in design and development time: these are just a few of the benefits that LLMs promise to offer to conversational AI teams. However, to unlock these, organizations usually need to go through the process of finding the right technological infrastructure to support them.

Evaluating and choosing the right conversational AI tool to suit your needs is a very complex and sensitive process, as industry experts highlight. To guide you through it, we’ve created a comprehensive list of the 18 top criteria to take into consideration.

Today, we expand on those baseline CAI platform characteristics, to focus on what vendors can offer when it comes to LLM-related features. With the constant releasing of new functionalities, understanding what vendors offer what can be tricky.

With LLMs and generative AI becoming more and more relevant for conversational AI teams, the process of platform evaluation needs to reflect on these capabilities more closely.

As discussed in detail in our guide to finding the right CAI technology partner, one size does not fit all in this context. The process of platform selection starts always from an introspective analysis of the organization itself. Only once the current state of the company, as well as its goals, are clearly identified, can the selection of fitting requirements start. 

To help you sort through all of this, we’ve identified 5 areas that LLMs & generative AI capabilities can support:

• NLU/NLP training & testing
• Design & implementation
• Testing & analytics
• User interaction
• Human agent support

Aside from those, we’ll also be addressing some of the general features and settings you might want your technology partner to have (or not). These are not capabilities per se, but are worth paying attention to, to ensure you’re able to safely and effectively implement your vision.

NLU/NLP Training & Testing with LLMs

This is one of the areas that LLMs could be revolutionizing altogether. In very simple terms: instead of working on training intents, collecting entity synonyms and so on, the bots of the future might entirely rely on LLMs to identify user's intentions (see Zero-Shot Model in the Runtime section).

Today, traditional intent-based NLU is still very much prominent and will likely hold its place for a while, at least for companies operating in sensitive domains or for use cases with strict requirements. That doesn’t mean that the process of training and testing an NLU model can’t be aided by LLMs.

Two popular capabilities offered in support of NLU/NLP training and testing are:

• Few-shot learning, i.e. allowing the model to extrapolate underlying patterns from just a few examples of data. It is pretty common for CAI vendors today to offer training utterance suggestions, and generating entities synonyms, which a human AI trainer might then review and choose to add to the training data.
• Test case generation, proposing variations of the training data for intents and entities, to test the effectiveness of the model.

Conversation Design & Implementation with LLMs

The processes of conversation design and development can also be enhanced and sped up with the use of LLMs, with platforms offering:

• LLM blocks, or nodes, that can be added to a specific point of a flow. These blocks can either be used to entertain the user in a natural-flowing conversation, or be aimed at achieving a goal, which will need to be described in the block’s prompt.
• Flow generation, which allows designers to automatically generate a whole flow interaction from a description of the task. The design can, then, be altered, to best represent the team’s vision.
• Translation, to localize the contents in different languages and contexts.
• Code generation, allowing to translate natural language into code.

CAI Testing & Analytics with LLMs

Perhaps less commonly talked about, compared to the first two areas we’ve so far addressed, testing conversational AI and keeping track of interactions with customers can also benefit from LLM support, through:

• Generation of test case suggestions, to perform routine, quick tests of the designs.
• Sentiment analysis, to determine customer satisfaction and feelings towards the bot directly from their interactions.

User interaction: Customer-Facing LLM Applications

One of the most interesting as well as potentially concerning areas of CAI to be revolutionized by LLMs is the direct interaction with the customer. 

Whereas the capabilities described so far come into play behind the scenes, the following are splayed out at runtime, during the direct interaction with the customer:

• Answering FAQs from documents/URLs, so that you can simply upload your materials and the bot will pull from those to respond to common queries about your business.
• Contextual and personalized rewriting of bot messages, which authorizes the bot to rephrase its responses to match the user’s tone and additional comments, or requests. Some platforms also allow to carry out entity collection in a free-flowing conversation, so that the designers identify what is the information needed from the user, but the bot gathers it in a more natural way.
• Generation of fallbacks/reprompts, to encourage users to answer questions or carry on with the conversation with dynamic messages
• Zero-shot classification of intents, to determine the intent at run time, without need to rely on a fully trained NLU model

LLM-based Human Agent Support

Finally, some vendors offer LLM capabilities to cover areas of customer service outside of the Conversational AI. Some features supporting human agents include:

• Agent hand-over summary, to report to the human agent what the interaction with the bot has been about so far and get them up to speed.
• Writing aid for agents, offering suggestions of rephrasing or, even, options of contextual, pre-written answers to use as responses.

Evaluating General LLM-Related Features & Settings in CAI Vendors

So far, we’ve been discussing what LLM technologies enable teams to do when it comes to customer service automation. If you’re evaluating what CAI vendor you want to select based on those, there are also some other considerations to be made. 

The following relates to general features and settings to check on, to verify how trustworthy and customizable the vendor you’re considering is:

• Number & quality of supported LLMs. With new and improved LLMs coming out every other day, you want to make sure that your platform will be able to support you in staying up to date and deploying the latest, best models. Verifying how many and what kind of models are currently available will give you an idea not only of what your team will immediately be able to work with, but also of the vendor’s likelihood to keep advancing emergent models.
• Vendor’s confidence in promoting LLM features. Some platforms have been working harder than others on their features, but it might be difficult to spot the difference from the outside. Looking at who came out with it first will not do in this case. It might be more indicative to look at how many of the generative capabilities are still in beta, the amount and type of disclaimers, and the way they are presented. Warnings and skepticisms towards LLMs are warranted, but there should be some degree of confidence that the features they propose are actually usable.
• Ease of providing domain knowledge. Some vendors will allow upload documents and maybe URLs to give contextual knowledge about the company and its services, others will also directly connect to CRM/CMS. The latter can be a necessity for bigger organizations, especially from a long-term perspective.
• Adjustability of exposure. Relying on LLMs always comprises a degree of risk. However, it is preferable to be able to autonomously control to what extent you want to expose yourself to the dangers of hallucinations. To ensure this, check for the ability to adjust temperature settings and to only draw from domain knowledge to respond to user queries (without pulling, e.g., from unknown sources from the model itself or even from the internet).
• Design customizations. To make sure your team’s vision will be able to come to life and to further manage your risk exposure, it’s helpful to know you'll be able to personalize your prompts and designs. Some of the setting you might look for include:
  • Bot persona dashboard, or an editable system prompt
  • Editable use case prompts, to define how you expect the bot to interact in a specific situation
  • Providing examples along with the use case prompts, in the form of describing a sample end-to-end interaction between the user and the bot
  • Defining the number of sentences, or words, to be used in the output
  • Specifying how many of the previous user inputs should be considered as context in the framing of the output
  • Establishing output type (text, image, link, a mix of more types, etc.)

In sum

From the outset, it's clear that the landscape of conversational AI platforms is far from uniform, particularly when considering the capabilities reliant on LLMs and generative AI. The quest for the perfect fit for your business is a nuanced endeavor, as the effectiveness of these technologies varies depending on the specific attributes and demands of your company.

In the context of this article, we've considered the capabilities and features we consider to gain an exhaustive understanding of the strengths and limitations inherent in each available option. As highlighted from the beginning, the points brought forward here are focused on LLMs and generative AI and these considerations should be paired with more general ones, to make an informed decision about what vendor to partner with.

We understand that the process of evaluating different technology vendors can be overwhelming and complex. That's why we also offer strategic guidance and support to help businesses choose the best technology for their unique requirements.

‍

‍‍Looking to figure out how conversational AI can help your business achieve your desired outcomes and drive growth?

‍

Feel free to reach out via via voice message or good old email.